lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <>
Subject [jira] [Updated] (SOLR-11653) create next time collection based on a fixed time gap
Date Mon, 01 Jan 2018 16:35:02 GMT


David Smiley updated SOLR-11653:
    Attachment: SOLR-11653.patch

Updated patch, significantly moving this issue to being done.  I tested it which uncovered
various bugs which I fixed.  I tweaked some but not all metadata param names to align with
SOLR-11722; we can fully align them here or there.  Unfortunately there's a bit of redundant
stuff in that issue and this one (and internally in this one between the URP & Cmd) that
we can resolve in a separate issue to tidy up duplication.  Some reflection on changes in
this patch:

Overseer & OverseerTaskProcessor: added checks for "operation" being null to pass a clear
error.  Of course this should never happen in bug free code, but my code wasn't bug free at
first :-).  In OTP I left a nocommit because I think the order of the nearby code should be
reordered a little (a very minor refactoring). The patch won't easily show this as there isn't
enough contextual lines of code.  I think operation should be gotten & verified (not null),
then check for the asyncId logic, and then acquire the messageHandler (thus immediately before
the MH is used).

CollectionsHandler.waitForActiveCollection: I made this method public because it's important
to check for this level of collection readiness before we actually use it.  I had hoped waitForFinalState
would be sufficient but tests proved it wasn't.  Having to call this feels bad; I have to
wonder if Collection creation should internally wait for being "active"; what's the harm in

IMO CollectionsHandler.handleResponse is a bad name... it's name suggest it only handles the
response when it performs the intended action too!  I have TODOs to rename to submitToOverseerRPC
since it submits to the Overseer queue and it does so in an RPC way (expects a response; it's
not just a message on a queue).  Input on the name is welcome.

Added TZ timezone handling for working with the interval.

What I want to do now is improve testing for race conditions by having the test submit documents
in parallel.  I may also get to a minor TODO involving the addition of a Lock to reduce wasted
concurrent calls by the URP to the Overseer to create a new collection.  This is minor because
the RoutedAliasCreateCollectionCmd doesn't do anything if ifHeadCollName doesn't jive with
the actual head collection name.

[~gus_heck] can you please review this now?

> create next time collection based on a fixed time gap
> -----------------------------------------------------
>                 Key: SOLR-11653
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>            Reporter: David Smiley
>            Assignee: David Smiley
>         Attachments: SOLR-11653.patch, SOLR-11653.patch
> For time series collections (as part of a collection Alias with certain metadata), we
want to automatically add new collections. In this issue, this is about creating the next
collection based on a configurable fixed time gap.  And we will also add this collection synchronously
once a document flowing through the URP chain exceeds the gap, as opposed to asynchronously
in advance.  There will be some Alias metadata to define in this issue.  The preponderance
of the implementation will be in TimePartitionedUpdateProcessor or perhaps a helper to this
> note: other issues will implement pre-emptive creation and capping collections by size.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message