lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shalin Shekhar Mangar (JIRA)" <>
Subject [jira] [Commented] (SOLR-5872) Eliminate overseer queue
Date Tue, 18 Mar 2014 08:09:43 GMT


Shalin Shekhar Mangar commented on SOLR-5872:

bq. as we move the individual collection states out of the main clusterstate.json [...]

This will make a difference on clusters with many smaller collections, but not on the single
big collection.
It seems like we still want scalability in both directions (wrt number of collections, and
the size a single collection can be).

The best solution that I see here is to move the replica states out into their own ZK nodes.
This way the individual nodes can update them directly without the overseer via compare and
set operations. The rest of the operations can continue to be processed in the overseer. If
we do this, even the external collection changes may not be required. The leader information
can also be read directly from the leader election nodes.

> Eliminate overseer queue 
> -------------------------
>                 Key: SOLR-5872
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>          Components: SolrCloud
>            Reporter: Noble Paul
>            Assignee: Noble Paul
> The overseer queue is one of the busiest points in the entire system. The raison d'ĂȘtre
of the queue is
>  * Provide batching of operations for the main clusterstate,json so that state updates
are minimized 
> * Avoid race conditions and ensure order
> Now , as we move the individual collection states out of the main clusterstate.json,
the batching is not useful anymore.
> Race conditions can easily be solved by using a compare and set in Zookeeper. 
> The proposed solution  is , whenever an operation is required to be performed on the
clusterstate, the same thread (and of course the same JVM)
>  # read the fresh state and version of zk node  
>  # construct the new state 
>  # perform a compare and set
>  # if compare and set fails go to step 1
> This should be limited to all operations performed on external collections because batching
would be required for others 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message