lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Noble Paul (JIRA)" <>
Subject [jira] [Commented] (SOLR-5872) Eliminate overseer queue
Date Tue, 18 Mar 2014 02:53:44 GMT


Noble Paul commented on SOLR-5872:

bq.I'm not really fully up on "external collections" but AFAIK it's part of some other work
to support tons of collections

There is more and more information getting added to the cluster state . I'm sure no one would
object to the point that splitting the clusterstate.json would be a more scalable solution
and the right direction to take. Of course, this is not to be done in haste , but eventually
that should be the way.  The eventual goal should be to support very large no:of collections
(say 1000's) and support extremely large collections (with 1000's of slices) . Solr itself
will not have any problem scaling like that but the Overseer/clusterstate strategy will go
through a revamp before we reach there.  

> Eliminate overseer queue 
> -------------------------
>                 Key: SOLR-5872
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>          Components: SolrCloud
>            Reporter: Noble Paul
>            Assignee: Noble Paul
> The overseer queue is one of the busiest points in the entire system. The raison d'ĂȘtre
of the queue is
>  * Provide batching of operations for the main clusterstate,json so that state updates
are minimized 
> * Avoid race conditions and ensure order
> Now , as we move the individual collection states out of the main clusterstate.json,
the batching is not useful anymore.
> Race conditions can easily be solved by using a compare and set in Zookeeper. 
> The proposed solution  is , whenever an operation is required to be performed on the
clusterstate, the same thread (and of course the same JVM)
>  # read the fresh state and version of zk node  
>  # construct the new state 
>  # perform a compare and set
>  # if compare and set fails go to step 1
> This should be limited to all operations performed on external collections because batching
would be required for others 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message