lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Gerlowski <gerlowsk...@gmail.com>
Subject Re: CloudSolrClient getDocCollection
Date Fri, 08 Feb 2019 22:20:40 GMT
Hi Henrik,

I'll try to answer, and let others correct me if I stray.  I wasn't
around when CloudSolrClient was written, so take this with a grain of
salt:

"Why does the client need that timeout?....Wouldn't it make sense to
use a watch?"

You could probably write a CloudSolrClient that uses watch(es) to keep
track of changing collection state.  But I suspect you'd need a
watch-per-collection, instead of just a single watch.

Modern versions of Solr store the state for each collection in
individual "state.json" ZK nodes
("/solr/collections/<collection_name>/state.json").  To catch changes
to all of these collections, you'd need to watch each of those nodes.
Which wouldn't scale well for users who want lots of collections.  I
suspect this was one of the concerns that nudged the author(s) to use
a cache-based approach.

(Even when all collection state was stored in a single ZK node, a
watch-based CloudSolrClient would likely have scaling issues for the
many-collection use case.  The client would need to recalculate its
state information for _all_ collections any time that _any_ of the
collections changed, since it has no way to tell which collection was
changed.)

Best,

Jason

On Thu, Feb 7, 2019 at 11:44 AM Hendrik Haddorp <hendrik.haddorp@gmx.net> wrote:
>
> Hi,
>
> when I perform a query using the CloudSolrClient the code first
> retrieves the DocCollection to determine to which instance the query
> should be send [1]. getDocCollection [2] does a lookup in a cache, which
> has a 60s expiration time [3]. When a DocCollection has to be reloaded
> this is guarded by a lock [4]. Per default there are 3 locks, which can
> cause some congestion. The main question though is why does the client
> need that timeout? According to this [5] comment the code does not use a
> watch. Wouldn't it make sense to use a watch? I thought the big
> advantage of the CloudSolrClient is that is knows were to send requests
> to, so that no extra hop needs to be done on the server side. Having to
> query ZooKeeper though for the current state does however take some of
> that advantage.
>
> regards,
> Hendrik
>
> [1]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L849
> [2]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1180
> [3]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L162
> [4]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1200
> [5]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L821

Mime
View raw message