lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gus Heck (JIRA)" <>
Subject [jira] [Commented] (SOLR-13439) Make collection properties easier and safer to use in code
Date Tue, 07 May 2019 14:51:00 GMT


Gus Heck commented on SOLR-13439:

{quote}By using only {{collectionWatches}}, the issue I mentioned in SOLR-13420 will happen.
Argh, that's wrong. I definitely meant to use {{collectionPropsWatches}} not {{collectionWatches}}.
Probably an auto-complete typo. I'm sure I fell into the trap of being unable to see it because
I knew what it *should* say thereafter. Thanks for catching this! 
{quote}you seem to be fetching twice here.
True, I'll need to adjust refreshAndWatch() to have a return value, but I see no reason not
to do that
{quote}It would be nice to have tests for {{ConditionalExpiringCache}}
Yes, :) I'll do that, though it's position in the RA URP causes some paths at least to be
tested in almost every test test so far. The test will of necessity be @Slow annotated anyway...

As for expiring vs un-register, its a trade off that favors long term stability and ease of
use at the cost of a thread and some background cpu (and a small timestamp update during access).
Register/unregister is unusable without a built in lifecycle on the client side, and makes
it very easy to write code that fails to unregister in the event of an error/exception, or
programmer mistake. Expiration of abandoned resources is more easily guaranteed than unregistration
at the end of usage... similar to the reasons that web application sessions don't rely on
users to log out and the JVM collects unused objects.

> Make collection properties easier and safer to use in code
> ----------------------------------------------------------
>                 Key: SOLR-13439
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>    Affects Versions: master (9.0)
>            Reporter: Gus Heck
>            Assignee: Gus Heck
>            Priority: Major
>         Attachments: SOLR-13439.patch
> (breaking this out from SOLR-13420, please read there for further background)
> Before this patch the api is quite confusing (IMHO):
>  # any code that wanted to know what the properties for a collection are could call zkStateReader.getCollectionProperties(collection)
but this was a dangerous and trappy API because that was a query to zookeeper every time.
If a naive user auto-completed that in their IDE without investigating, heavy use of zookeeper
would ensue.
>  # To "do it right" for any code that might get called on a per-doc or per request basis
one had to cause caching by registering a watcher. At which point the getCollectionProperties(collection) magically
becomes safe to use, but the watcher pattern probably looks famillar induces a user who hasn't
read the solr code closely to create their own cache and update it when their watcher is
notified. If the caching side effect of watches isn't understood this will lead to many in-memory
copies of collection properties maintained in user code.
>  # This also creates a task to be scheduled on a thread (PropsNotification) and induces
an extra thread-scheduling lag before the changes can be observed by user code.
>  # The code that cares about collection properties needs to have a lifecycle tied to
either a collection or something other object with an even more ephemeral life cycle such
as an URP. The user now also has to remember to ensure the watch is unregistered, or there
is a leak.
> After this patch
>  # Calls to getCollectionProperties(collection) are always safe to use in any code anywhere.
Caching and cleanup are automatic.
>  # Code that really actually wants to know if a collection property changes so it can
wake up and do something (autoscaling?) still has the option of registering a watcher that
will asynchronously send them a notification.
>  # Updates can be observed sooner via getCollectionProperties with no need to wait for
a thread to run. (vs a cache held in user code)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message