solr(8)Solr Index Replication

solr(8)Solr Index Replication

1. Introduction Index Replication
The master will handle updates to the index, all querying is handled by the slaves.

Master — Slave1, Slave2, Slave3. A Solr index can be replicated across multiple slave servers, which then process requests.

Index Replication in Solr
Solr includes a Java implementation of index replication that works over HTTP.
solrconfig.xml, works across platforms with same configuration.

Some Term
Index - A lucene index is a directory of files. These files make up the searchable and returnable data of a Solr Core.
Distribution - The copying of an index from the master server to all slaves.
Inserts and Deletes - The directory remains unchanged. Documents are always inserted into newly created files. Documents
that are deleted are not removed from the files. They are flagged in the file, deletable, and are not
removed from the files until the index is optimized.
Master and Slave - Slave nodes receive no updates directly, instead all changes are made against the single master node.
Changes made on the master are distributed to all the salve nodes.
Update -
Optimization - Optimization should only be run on the master nodes.
Segment -
mergeFactor -
Snapshot -

Configuring the ReplicationHandler
maxNumberOfBackups - how many backups we keep on disk as it receives back commands.

Configuring the Replication RequestHandler on a Master Server
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
<str name="replicateAfter">optimize</str>
<str name="backupAfter">optimize</str>
<str name="confFiles">schema.xml,stopwords.txt,elevate.xml</str>
<str name="commitReserveDuration">00:00:10</str>
</lst>
<int name="maxNumberOfBackups">2</int>
<lst name="invariants">
<str name="maxWriteMBPerSec">16</str>
</lst>
</requestHandler>

Configuring the Replication RequestHandler on a Slave Server
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="slave">
<!-- fully qualified url for the replication handler of master. It is
possible to pass on this as a request param for the fetchindex command -->
<str name="masterUrl">http://remote_host:port/solr/core_name/replication</str>
<!-- Interval in which the slave should poll master. Format is HH:mm:ss .
If this is absent slave does not poll automatically.
But a fetchindex can be triggered from the admin or the http API -->
<str name="pollInterval">00:00:20</str>
<!-- THE FOLLOWING PARAMETERS ARE USUALLY NOT REQUIRED-->
<!-- To use compression while transferring the index files. The possible
values are internal|external. If the value is 'external' make sure
that your master Solr has the settings to honor the accept-encoding header.
See here for details: http://wiki.apache.org/solr/SolrHttpCompression
If it is 'internal' everything will be taken care of automatically.
USE THIS ONLY IF YOUR BANDWIDTH IS LOW.
THIS CAN ACTUALLY SLOWDOWN REPLICATION IN A LAN -->
<str name="compression">internal</str>
<!-- The following values are used when the slave connects to the master to
download the index files. Default values implicitly set as 5000ms and
10000ms respectively. The user DOES NOT need to specify these unless the
bandwidth is extremely low or if there is an extremely high latency -->
<str name="httpConnTimeout">5000</str>
<str name="httpReadTimeout">10000</str>
<!-- If HTTP Basic authentication is enabled on the master, then the slave
can be configured with the following -->
<str name="httpBasicAuthUser">username</str>
<str name="httpBasicAuthPassword">password</str>
</lst>
</requestHandler>

Setting Up a Repeater with the ReplicationHandler
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
<str name="replicateAfter">commit</str>
<str name="confFiles">schema.xml,stopwords.txt,synonyms.txt</str>
</lst>
<lst name="slave">
<str name="masterUrl">http://master.solr.company.com:8983/solr/core_name/replication</str>
<str name="pollInterval">00:00:60</str>
</lst>
</requestHandler>

Commit and Optimize Operations

Slave Replication

Replicating Configuration Files

Reading the Sample Configuration and Setup an Replication Myself

2. Understanding of Solr
There is a URL/handlers mapping in solrconfig.xml.

Some built-in handlers in Solr
Search handlers: DisMaxRequestHandler, LukeRequestHandler, MoreLikeThisHandler, SearchHandler, SpellCheckerRequestHandler

We are using solr.SearchHandler, http://wiki.apache.org/solr/SearchHandler

Update handlers: DataImportHandler, BinaryUpdateRequestHandler, CSVUpdateRequestHandler, ExtractingRequestHandler, JsonUpdateRequestHandler, XmlUpdateRequestHandler, XsltUpdateRequestHandler

We are using
solr.RealTimeGetHandler,
solr.UpdateRequestHandler,
solr.ExtractingRequestHandler,
solr.FieldAnalysisRequestHandler,
solr.DocumentAnalysisRequestHandler,
solr.admin.AdminHandler,
solr.PingRequesthandler,
solr.DumpRequestHandler,
solr.ReplicationHandler,
solr.DirectUpdateHandler2

UpdateRequestProcessor and Chain, something like FilterChain. The implement class will extends from SearchComponent

Java - solrj
scala - https://github.com/inoio/solrs
https://github.com/takezoe/solr-scala-client

3. Configuration on Ubuntu Master and Slaves
…todo...


References:
solr 6 and solr 7
http://sillycat.iteye.com/blog/2227066
http://sillycat.iteye.com/blog/2227398

pull mode index replication
https://cwiki.apache.org/confluence/display/solr/Index+Replication

distributed search with index sharding
https://cwiki.apache.org/confluence/display/solr/Distributed+Search+with+Index+Sharding

Solr Cloud
https://cwiki.apache.org/confluence/display/solr/SolrCloud

Solr Article
http://blog.youkuaiyun.com/jaynol/article/details/17230857
http://blog.youkuaiyun.com/jaynol/article/details/24717123
http://blog.youkuaiyun.com/jaynol/article/details/24776437
http://blog.youkuaiyun.com/jaynol/article/details/24959373
http://blog.youkuaiyun.com/jaynol/article/details/25098667
http://blog.youkuaiyun.com/jaynol/article/details/25305323
http://blog.youkuaiyun.com/jaynol/article/details/25604271

lucene
https://lucene.apache.org/core/5_2_1/

solr clients
https://wiki.apache.org/solr/IntegratingSolr

XML for Solr
https://wiki.apache.org/solr/UpdateXmlMessages#XML_Messages_for_Updating_a_Solr_Index
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers

akka system
http://sillycat.iteye.com/blog/1767866
http://sillycat.iteye.com/blog/1768625
http://sillycat.iteye.com/blog/1768626
http://sillycat.iteye.com/blog/2099267
http://sillycat.iteye.com/blog/2100232

actor waiting
http://stackoverflow.com/questions/3107286/wait-for-an-actor-to-exit