lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Timothy Potter (JIRA)" <>
Subject [jira] [Updated] (SOLR-7333) Make the poll queue time configurable and use knowledge that a batch is being processed to poll efficiently
Date Wed, 01 Apr 2015 14:59:53 GMT


Timothy Potter updated SOLR-7333:
    Attachment: SOLR-7333.patch

Here's a first pass at a patch that uses a 25ms pollQueueTime when processing a batch of documents.
The main idea here is that the javabin unmarshalling code can detect when it sees the last
doc in a batch and can pass that hint down the line, eventually to the ConcurrentUpdateSolrServer
the leader uses to stream docs to replicas. CUSS uses that hint to poll the queue for 0 (vs.
waiting the 25ms). This helps stream more docs from the leader to replica per request, which
keeps the requests processed by leaders and replicas nearly the same and reduces round-trips
per batch between leader and replica. The hint about being the last doc in a batch is necessary
to avoid the wait when processing docs one-by-one or when the last doc in a batch has been
processed, i.e. poll the queue with a brief wait if more docs are available but don't wait
if not.

Currently, the pollQueueTime is hardcoded to 25 ms, but I suppose we could make that configurable.
The key is to use a short wait so I felt 25 ms should be sufficient for most indexing applications.

Lastly, I added the {{isLastDocInBatch}} flag as a member to UpdateRequest instead of including
it into the params because CUSS checks for params changing while processing UpdateRequests
in a batch and treats a change in parameters as a separate request, which is what we're trying
to avoid here.

> Make the poll queue time configurable and use knowledge that a batch is being processed
to poll efficiently
> -----------------------------------------------------------------------------------------------------------
>                 Key: SOLR-7333
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Timothy Potter
>            Assignee: Timothy Potter
>         Attachments: SOLR-7333.patch
> {{StreamingSolrClients}} uses {{ConcurrentUpdateSolrServer}} to stream documents from
leader to replica, by default it sets the {{pollQueueTime}} for CUSS to 0 so that we don't
impose an unnecessary wait when processing single document updates or the last doc in a batch.
However, the downside is that replicas receive many more update requests than leaders; I've
seen up to 40x number of update requests between replica and leader.
> If we're processing a batch of docs, then ideally the poll queue time should be greater
than 0 up until the last doc is pulled off the queue. If we're processing a single doc, then
the poll queue time should always be 0 as we don't want the thread to wait unnecessarily for
another doc that won't come.
> Rather than force indexing applications to provide this optional parameter in an update
request, it would be better for server-side code that can detect whether an update request
is a single document or batch of documents to override this value internally, i.e. it'll be
0 by default, but since {{JavaBinUpdateRequestCodec}} can determine when it's seen the last
doc in a batch, it can override the pollQueueTime to something greater than 0.
> This means that current indexing clients will see a boost when doing batch updates without
making any changes on their side.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message