lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gus Heck (JIRA)" <>
Subject [jira] [Commented] (SOLR-11653) create next time collection based on a fixed time gap
Date Tue, 02 Jan 2018 01:10:00 GMT


Gus Heck commented on SOLR-11653:

Some thoughts: 

It seems that you've added ROUTEDALIAS_CREATECOLL as a user accessible command (the v1 API
will respond to it I think) but not reflected in v2 api json files. I think this is because
this command is  probably not meant for API invocation in the first place, so it kinda looks
out of place as an undocumented API. I kind of wonder why it was done as an admin command.
I'm probably missing something, but it seems like this means we have:
# doc arrives for which it is appropriate to create a new collection.
# issue admin command ROUTEDALIAS_CREATECOLL and wait for it
# inside ROUTEDALIAS_CREATECOLL issue CollectionsHandler.CollectionOperation.CREATE_OP.execute(...
and wait for that
# finally process the update

I'm not sure why we want the extra layer? is there actually a use case for manual creation
of the next partition? To me it seems as if this operation is internal to TimeRoutedAliasUpdateProcessor
and should be there. I can possibly imagine that if we were proactively keeping ahead of things
by one collection this structure could allow fast processing of the update by giving it an
async id, but it looks like the code is set up to add a collection the first time it gets
a document that requires that collection (up to maxFutureMs), and delay the update until that

This makes me wonder if we should even be supporting +1SECOND? If this command is not sub-second
we fall behind... That probably also is a really bad idea for sheer number of collections
too. As a side benefit we could also shorten our collection names too...

I think your loop for creating collections should be dependent on the value of maxFutureMs?
Let's say now() is 01:22 the head is presently ending in _01_00 and accepting 01:00 hrs to
02:00 hrs, maxFutureMs=72*60*60*1000 and we get a document for 60 hours in the future... looks
like we loop up to 5 times and successfully create the _02_00, 0_03_00, _04_00,_05_00,_06_00
partitions and then error out unless we get an update to our parsedCollectionAliases in the
mean time... if any 5 creations are faster than the updates to parsedCollectionAliases we
wrongly error out. The error message will be misleading too since the attempts to create the
collections all succeeded but we quit due to some sort of communications lag. IF the parsedCollectionsAliases
gets updated in time we will reset our counter and keep going, but why not directly calculate
the appropriate minimum number of attempts from DocTimestampMs - HeadTimestampMs divided by
HeadTimestampMs+(one interval) - HeadTimestampMs?  A constant can be added if we want some
slack for retrying failed calls.

Perhaps my work in SOLR-11722 should be filling out all prospective next collections up to
maxFutureMs (which I need to add to metadata)? Then in this code rather than testing the current
document's time stamp to see if we create a new collection, test whether or not the next increment
falls within MaxFutureMS. In the case that it's time to create a new collection we can then
check the document value and create the next one asynchronously unless the current document
would fall in the yet to be created collection (which should normally be a rare case). That
would go from one guaranteed slow update every hour (for +1HOUR routed aliases) to only having
a slow update if a document at the very beginning of an hour happens to fall *just* inside
maxFutureMs. We would need to track that the collection was in progress for creation of course
to avoid spamming the overseer each hour... 

It seems easier and more robust to never need to create more than one collection during an
update. In that case your present loop is just fine.

In code, you have variable names and comments talking about "head" but I had to dig a little
to confirm that this was actually the "most recent" not "the first".... be nice if comments
made this clear.

I agree with your desire to rename handleResponse() ... confused me at first.

> create next time collection based on a fixed time gap
> -----------------------------------------------------
>                 Key: SOLR-11653
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>            Reporter: David Smiley
>            Assignee: David Smiley
>         Attachments: SOLR-11653.patch, SOLR-11653.patch
> For time series collections (as part of a collection Alias with certain metadata), we
want to automatically add new collections. In this issue, this is about creating the next
collection based on a configurable fixed time gap.  And we will also add this collection synchronously
once a document flowing through the URP chain exceeds the gap, as opposed to asynchronously
in advance.  There will be some Alias metadata to define in this issue.  The preponderance
of the implementation will be in TimePartitionedUpdateProcessor or perhaps a helper to this
> note: other issues will implement pre-emptive creation and capping collections by size.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message