lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-7147) Intoduce new TrackingShardHandlerFactory for monitoring what requests are sent to shards during tests
Date Tue, 24 Feb 2015 00:43:11 GMT

    [ https://issues.apache.org/jira/browse/SOLR-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14334171#comment-14334171
] 

Hoss Man commented on SOLR-7147:
--------------------------------

bq. I created a TrackingShardHandlerFactory which can record shard requests sent from any
node. There are a few helper methods to get requests by shard and by purpose.

I like the idea ... hooking into the ShardHandlerFactory seems like a good solution to these
types of problems...

bq. I will likely move the TrackingShardHandlerFactory into its own issue because it is helpful
for other distributed tests as well.

Yeah, a top level class in the test-framework would be good, but...

I'm not sure if a {{Map<String, List<ShardRequestAndParams>>}} is really the best
solution here?

It's good enough for this small test case, but as a general solution ...  consider Test Classes
were folks want to setup their distrib cluster, do a handfull of specific queries to monitor
what gets sent to each shard, and then in other test methods crank out thousands of distrib
queries.  the way this is setup, that's either going to eat up a ton of ram, or the test is
going to eat up a ton of ram, or the test is going to need to be sprinkled with tons of calls
to some clearOutShardHandlerTrackers() utility method

perhaps instead, the TrackingShardHandlerFactory should have a setTrackingQueue/getTrackingQueue
methods for a {{Queue<ShardRequestAndParams>}} (and ShardRequestAndParams should also
have a {{String shard}} property - although isn't this already in the ShardRequest?).

By default, the Queue is null and TrackingShardHandlerFactory doesn't track anything -- but
if it's non-null, then the ShardHandler calls {{add(new ShardRequestAndParams(...)}} just
before passing executing it.

this way, there is almost no overhead to using TrackingShardHandlerFactory unless/until a
test explicitly specifies a Queue.  Test classes, for the portions of the test when they want
to monitor shard requests, could call setTrackingQueue on a custom subclass of AbstractQueue
that directly asserts things like...
{code}
if (req.sreq.purpose & PURPOSE_GET_TOP_IDS) {
  assertEquals(expected, req.sreq.getParams(FL));
}
{code}
 .. alternatively, a LinkedBlockingQueue could be specified, and the test class could spin
up an Executor that was would poll that Queue continuously inspecting them & recording
info for processing/assert at the end of the test.

at any point when the test case no longer cares about tracking, setTrackingQueue(null) again.

The simplier use case of what you're doing here could still pretty easily be supported, just
by providing a simple special case Queue class that has a {{Map<String, List<ShardRequestAndParams>>
getAll()}} method on it, and tests could do something like...

{code}
  TrackAllQueue myQueue = TrackingShardHandlerFactory.setupTrackAllQueue(myMiniSolrCloudCluster,
collectionName);
  // .. do test stuff ..
  TrackingShardHandlerFactory.setAllTrackingQueues(myMiniSolrCloudCluster, null)
  List<ShardRequestAndParams> list = myQueye.getAll().get(coreUrl);
  // ...
{code}

bq. I will likely move the TrackingShardHandlerFactory into its own issue because it is helpful
for other distributed tests as well. I also need to decouple it from the MiniSolrCloudCluster
abstraction.

I think the main thing would be some static methods in TrackingShardHandlerFactory to get/set
the TrackingQueue for all JettySolrRunners in a list, or on a MiniSolrCloudCluster (in which
case it just pulls out the JettySolrRunners from there)

what do you think?


> Intoduce new TrackingShardHandlerFactory for monitoring what requests are sent to shards
during tests
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-7147
>                 URL: https://issues.apache.org/jira/browse/SOLR-7147
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Hoss Man
>
> this is an idea shalin proposed as part of the testing for SOLR-7128...
> bq. I created a TrackingShardHandlerFactory which can record shard requests sent from
any node. There are a few helper methods to get requests by shard and by purpose.
> ...
> bq. I will likely move the TrackingShardHandlerFactory into its own issue because it
is helpful for other distributed tests as well. I also need to decouple it from the MiniSolrCloudCluster
abstraction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message