lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramkumar Aiyengar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-5872) Eliminate overseer queue
Date Sat, 15 Aug 2015 20:27:45 GMT

    [ https://issues.apache.org/jira/browse/SOLR-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698435#comment-14698435
] 

Ramkumar Aiyengar commented on SOLR-5872:
-----------------------------------------

Though I haven't done serious experiments on this as yet, I see the lack of batching in stateFormat=2
is a potential blocker to it's adoption. We need some benchmarks on a single collection with
lots of cores (at least 1000), and see how it works with stateFormat=1, stateFormat=2, and
this new approach. Keep in mind that hundreds of cores might change state at the same time,
that's the real benefit to batching. I fear that without a batching approach, the system might
choke due to the contention at that point.

> Eliminate overseer queue 
> -------------------------
>
>                 Key: SOLR-5872
>                 URL: https://issues.apache.org/jira/browse/SOLR-5872
>             Project: Solr
>          Issue Type: Improvement
>          Components: SolrCloud
>            Reporter: Noble Paul
>            Assignee: Noble Paul
>
> The overseer queue is one of the busiest points in the entire system. The raison d'ĂȘtre
of the queue is
>  * Provide batching of operations for the main clusterstate,json so that state updates
are minimized 
> * Avoid race conditions and ensure order
> Now , as we move the individual collection states out of the main clusterstate.json,
the batching is not useful anymore.
> Race conditions can easily be solved by using a compare and set in Zookeeper. 
> The proposed solution  is , whenever an operation is required to be performed on the
clusterstate, the same thread (and of course the same JVM)
>  # read the fresh state and version of zk node  
>  # construct the new state 
>  # perform a compare and set
>  # if compare and set fails go to step 1
> This should be limited to all operations performed on external collections because batching
would be required for others 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message