Apache Solr with WSO2

Creating an Online Workspace: Integrating Apache Solr with
WSO2 ESB for Content Search

Online Workspaces

In today’s digital age, effective collaboration is increasingly achieved via an online workplace which provides a holistic set of workplace tools, platforms and environments, delivered in a coherent, usable and productive way.

Online workplaces help to empower individuals, teams, colleagues, clients and partners to share, communicate and collaborate with little to no friction, delay or challenge; allowing all these stakeholders to perform their jobs more effectively.
One of our clients recently expressed their need for an online workspace to keep track of business process documentations, data sheets, process workflows, media files etc. As systems integration experts, we ran this project knowing that we could effectively build the workspace using Apache Solr and WSO2 ESB.

Because we love to share knowledge, this ‘How-To-Guide’ outlines how Apache Solr can be successfully integrated for search within applications, using WSO2 ESB.


Apache Solr

Apache Solr is an open source search platform built upon a Java library called Lucene. It is a popular search platform for web sites because it can index and search multiple sites and return recommendations for related content based on the search query’s taxonomy. Apache Solr is also a popular search platform for enterprise search because of its ability to index, search documents, and email attachments.
It has a range of versatile features which include:

● Advanced full-text search capabilities
● Standards based open interfaces – XML, JSON and HTTP
● Comprehensive administration interfaces
● Easy monitoring
● Highly scalable and fault tolerant
● Flexible and adaptable with easy configuration
● Near real-time indexing
● Extensible plugin architecture
● Supports indexing and searching multiple sites
● Indexing attachments (Email, PDF)
● It is also optimised for high volume traffic.

Companies like Apple, AOL, Digg, Cnet, Disney and Netflix use Apache Solr for their search feature.

WSO2 ESB

WSO2 Enterprise Service Bus (WSO2 ESB) is a lightweight, high performance, comprehensive ESB that is 100% open source. WSO2 ESB effectively addresses integration standards and supports all integration patterns, enabling interoperability among various heterogeneous systems and business applications.

Its intuitive visually supported tools offer users a smooth experience in development environments and production deployments. Designed for customisation, WSO2 ESB incorporates ESB specific analytics dashboards to gain better insights into system performance.

Equipped with drill-down capabilities, it also allows users to instantly understand and trace mediation flows for better analysis.

WSO2 ESB provides greater flexibility and agility to meet growing enterprise demands.

Integrating WSO2 ESB with Apache Solr Server

We took our first project step by building and implementing an Atlassian Confluence-like software application to hold documentation for the client’s business processes. We built it using the Apache Solr Server to enable content publishing, the creation of indexes, and full text searches.

The application we created allows a logged user to go to a particular library within the system, look for relevant business process documents and read or print them. When a user types a word or phrase into the system to access a particular document, the search results appear in a list – similar to Google’s search results – with highlighted texts.
We also added some additional functionalities. These included:

● Frequent database syncing
● Tenant wise content filtering
● Role based content filtering.

With regards to publishing dynamic properties and indexing in the Solr server, we faced some challenges. This is because we initially decided to use an NOSQL Graph Database (Neo4J) to store documents in the system.

Using an NOSQL database can lead to problems because, unlike the RDBMS table, different properties can exist in the database for the same entity type, thus, the Indexer needs to be able to Index unique dynamic properties based on both Tenant and Roles.

It was time to introduce WSO2 ESB to the project!

Content Search using WSO2 ESB

Once the indexer was created to publish, we knew that WSO2 ESB middleware would be the best content search solution, as the search requests coming from the web can be intercepted by the ESB and modified according to user privileges. They are then submitted to the Solr server through the Apache Solr REST API, and structured in such a way that they populate back into the web in a similar fashion to Google search results.

The image below provides an overview of the solution we developed:

Content Search using WSO2 ESB

The role of WSO2 ESB in Search:
● It can accept search requests from the website: These requests contain a search phrase and JWT Authentication header (an authentication mechanism for server communication) comprising of the Organisation ID (Tenant) and roles
● It intercepts the request and builds a Solr search request using search phrases, Organisation ID, and roles
● It calls the Apache Solr Rest API and waits for a response
● It receives the response and builds the web response accordingly.
results.

Outcome and Milestones to be Accomplished

Integrating Apache Solr with WSO2 ESB for Content Search helped us to fulfill the requirements for the project. Therefore we aim to write a fully fledged Apache Solr Connector to WSO2 ESB in the near future, which can support dynamic properties, user privileges and roles as well as develop generic content searches and customisable results.

Conclusion

As experts in systems integration, we love to find ways to get existing and new technologies to communicate with each other across various platforms to achieve a common aim. And using powerful WSO2 middleware, we are able to succeed on integration projects.

Integrating WSO2 ESB with Apache Solr was a good plan because the middleware is designed for customisation, effectively addresses integration standards, supports all integration patterns and enables interoperability among various heterogeneous systems and business applications.

We hope you’ve found this ‘How-To-Guide’ informative. If you’d like to know more about WSO2 ESB and Apache Solr, please get in touch with us.

We also encourage software developers, entrepreneurs and enterprises to capitalise on WSO2 middleware capabilities to achieve your own goals.

Lakshitha herath

About the Author

Lakshitha Herath is a Technical Lead at Mitra Innovation, a technology company that specialises in product and company incubation, systems integration, and digital innovation and transformation.

Find out how Lakshitha and the rest of the Mitra Innovation team can help you make the most out of the Apache Solr platform with WSO2 ESB.