lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <>
Subject [jira] [Commented] (SOLR-12357) TRA: Pre-emptively create next collection
Date Thu, 06 Sep 2018 06:12:00 GMT


David Smiley commented on SOLR-12357:

It sort of appears that the TrackingUpdateProcessorFactory test utility was buggy in some
way, though I'm not sure how exactly.  It seemed more complex than how I reasoned in my mind
it needs to be to do its job, so I wound up simplifying it, as seen in the attached patch.
 This passes so far; in the morning my beasting will be done but I need to sleep as I write

> TRA: Pre-emptively create next collection 
> ------------------------------------------
>                 Key: SOLR-12357
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>            Reporter: David Smiley
>            Priority: Major
>         Attachments: SOLR-12357.patch
>          Time Spent: 9h 20m
>  Remaining Estimate: 0h
> When adding data to a Time Routed Alias (TRA), we sometimes need to create new collections.
 Today we only do this synchronously – on-demand when a document is coming in.  But this
can add delays as the documents inbound are held up for a collection to be created.  And,
there may be a problem like a lack of resources (e.g. ample SolrCloud nodes with space) that
the policy framework defines.  Such problems could be rectified sooner rather than later
assume there is log alerting in place (definitely out of scope here).
> Pre-emptive TRA collection needs a time window configuration parameter, perhaps named
something like "preemptiveCreateWindowMs".  If a document's timestamp is within this time
window _from the end time of the head/lead collection_ then the collection can be created
pre-eptively.  If no data is being sent to the TRA, no collections will be auto created,
nor will it happen if older data is being added.  It may be convenient to effectively limit
this time setting to the _smaller_ of this value and the TRA interval window, which I think
is a fine limitation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message