lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Per Steffensen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-4114) Collection API: Allow multiple shards from one collection on the same Solr server
Date Tue, 11 Dec 2012 15:43:22 GMT

     [ https://issues.apache.org/jira/browse/SOLR-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Per Steffensen updated SOLR-4114:
---------------------------------

    Attachment: SOLR-4114_mocking_OverseerCollectionProcessorTest_branch_4x.patch

I promissed Robert Muir to make a test of the feature introduced here in SOLR-4114 as a unit-test
directly on OverseerCollectionProcessor. I did this in attached SOLR-4114_mocking_OverseerCollectionProcessorTest_branch_4x.patch.
It fits on top of revision 1420194 of branch_4x, but shouldnt be hard to port to other branches,
since it is basically just a new test OverseerCollectionProcessorTest.

Besides the new test, OverseerCollectionProcessor has been modified a little in order to easily
be able to extend it in the test.

OverseerCollectionProcessorTest tests OverseerCollectionProcessor alone, by mocking the components
it interacts with directly:
* DistributedQueue - the work-queue with messages from ZK
* ZkStateReader
* ClusterState
* ShardHandler - the component handling/distributing the CoreAdmin requests comming out of
OverseerCollectionProcessor.

I wanted to use mockito but found that you are already using easymock, so I decided to use
that. I had to upgrade easymock from version 2.0 to version 3.0, because I wanted to mock
classes (not only interfaces) - nothing is interfaces in Solr. Guess no one would mind that.

OverseerCollectionProcessorTest tests a few things including the feature introduced here in
SOLR-4114, and to some extend eliminates the additional test-parts added to BasicDistributedZkTest
here in SOLR-4114. A.o. the controversial 10-60 sec wait test
{code}
int liveNodes = getCommonCloudSolrServer().getZkStateReader().getClusterState().getLiveNodes().size();
int numShards = (liveNodes/2) + 1;
int numReplica = 1;
int maxShardsPerNode = 1;
collectionInfos = new HashMap<String,List<Integer>>();
createCollection(collectionInfos, cnt, numShards, numReplica, maxShardsPerNode);
checkCollectionIsNotCreated(collectionInfos.keySet().iterator().next());
{code}

OverseerCollectionProcessorTest establishes a nice platform for testing OverseerCollectionProcessor
on unit-level using mocking, and can probably be extended to further eliminate tests in BasicDistributedZkTest.testCollectionAPI.
And it can be extended to do more than just "create"-tests - also do "reload"-tests and "remove"-tests.
                
> Collection API: Allow multiple shards from one collection on the same Solr server
> ---------------------------------------------------------------------------------
>
>                 Key: SOLR-4114
>                 URL: https://issues.apache.org/jira/browse/SOLR-4114
>             Project: Solr
>          Issue Type: New Feature
>          Components: multicore, SolrCloud
>    Affects Versions: 4.0
>         Environment: Solr 4.0.0 release
>            Reporter: Per Steffensen
>            Assignee: Per Steffensen
>              Labels: collection-api, multicore, shard, shard-allocation
>         Attachments: SOLR-4114_mocking_OverseerCollectionProcessorTest_branch_4x.patch,
SOLR-4114.patch, SOLR-4114.patch, SOLR-4114.patch, SOLR-4114.patch, SOLR-4114_trunk.patch
>
>
> We should support running multiple shards from one collection on the same Solr server
- the run a collection with 8 shards on a 4 Solr server cluster (each Solr server running
2 shards).
> Performance tests at our side has shown that this is a good idea, and it is also a good
idea for easy elasticity later on - it is much easier to move an entire existing shards from
one Solr server to another one that just joined the cluter than it is to split an exsiting
shard among the Solr that used to run it and the new Solr.
> See dev mailing list discussion "Multiple shards for one collection on the same Solr
server"

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message