Manage Solr Data in WSO2 Server

Recently I was checking an issue faced by one of my colleague while automating WSO2 API Manager deployment. There, once the new pack is deployed by pointing to the existing databases, APIM Store didn't show existing APIs at once. It took some time to display all the existing APIs in the Store.

The APIs are retrieved using the Solr based indexing in APIM. Therefore, the main reason for this behavior is that a fresh pack doesn't have existing Solr data and it takes some time to complete the indexing. Until that indexing process is completed, it will not show API in the Store instantly.

To address this, you can follow one of the below approaches:

1). Backup existing Solr data (APIM_HOME/solr/data) from the existing deployment and added it to newly created pack.


2). Externalize Solr data directory. Solr data stored location can be configured via core.properties file located in the APIM_HOME/repository/conf/solr/registry-indexing directory. So you can update core.properties to store Solr data outside the product directory.

Configure External Solr server with Governance Registry

In WSO2 Governance Registry 5.0.0, we have upgraded Apache Solr version into 5.2 release. With that you can connect WSO2 Governance Registry into an external Solr server or Solr cluster. External Solr integration provides features to gain comprehensive Administration Interfaces, High scalability and Fault Tolerance, Easy Monitoring and many more Solr capabilities.

Let me explain how you can connect WSO2 Governance Registry server with an external Apache Solr server.

1). First, you have to download Apache Solr 5.x.x from the below location.
http://lucene.apache.org/solr/mirrors-solr-latest-redir.html
Please note that we have only verified with Solr 5.2.0 and 5.2.1 versions only.

2). Then unzip Solr Zip file. Once unzipped, it's content will look like the below.

The bin folder contains the scripts to start and stop the server. Before starting the Solr server, you have to make sure JAVA_HOME variable is set properly. Apache Solr is shipped with an inbuilt Jetty server.

3). You can start the Solr server by issuing "solr start" command from the bin directory. Once the Solr server is started properly, following message will be displayed in the console. "Started Solr server on port 8983 (pid=5061). Happy searching!"

By default, server starts on port "8983" and you can access the Solr admin console by navigating to "http://localhost:8983/solr/".

4) To create a new Solr Core, you have to copy and paste Solr configuration directory(registry-indexing) found in G-REG_HOME/repository/conf/solr to SOLR_HOME/server/solr/ directory. Please note that only "registry-indexing" directory needs to be copied from the G-Reg pack.  This will create a new Solr Core named as "registry-indexing".

5). After creating "registry-indexing" Solr core, you can see it from the Solr admin console as below.

6). To integrate newly created Solr core with WSO2 Governance Registry, you have to modify registry.xml file located in <greg_home>/repository/conf directory. There you have to add "solrServerUrl" under indexingConfiguration as follows and need to comment out "IndexingHandler".

    <!-- This defines index cofiguration which is used in meta data search feature of the registry -->
  
    <indexingConfiguration>
        <solrServerUrl>http://localhost:8983/solr/registry-indexing</solrServerUrl> 
        <startingDelayInSeconds>35</startingDelayInSeconds>
        <indexingFrequencyInSeconds>3</indexingFrequencyInSeconds>
        <!--number of resources submit for given indexing thread -->
        <batchSize>50</batchSize>
         <!--number of worker threads for indexing -->
        <indexerPoolSize>50</indexerPoolSize>
        .................................
    </indexingConfiguration>

7). After completing external Solr configurations as above, you have to start the WSO2 Governance Registry server. If you have configured External Solr integration properly, you can notice below log message in the Governace Registry server startup logs(wso2carbon log).

[2015-07-11 12:50:22,306] INFO {org.wso2.carbon.registry.indexing.solr.SolrClient} - Http Sorl server initiated at: http://localhost:8983/solr/registry-indexing

Further, you can view indexed data by querying via Solr admin console as well.

Happy Indexing and Searching...!!!

Note(added on March 2016): If you are moving from older G-Reg version to the latest one,  you have to replace existing Solr Core(registry-indexing) with the latest one available in the G-Reg pack.