lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <>
Subject [jira] [Created] (SOLR-12729) SplitShardCmd should lock the parent shard to prevent parallel splitting requests
Date Mon, 03 Sep 2018 10:38:00 GMT
Andrzej Bialecki  created SOLR-12729:

             Summary: SplitShardCmd should lock the parent shard to prevent parallel splitting
                 Key: SOLR-12729
             Project: Solr
          Issue Type: Bug
      Security Level: Public (Default Security Level. Issues are Public)
          Components: AutoScaling
            Reporter: Andrzej Bialecki 
            Assignee: Andrzej Bialecki 
             Fix For: 7.5

This scenario was discovered by the simulation framework, but it exists also in the non-simulated

When {{IndexSizeTrigger}} requests SPLITSHARD, which is then successfully started and “completed”
from the point of view of {{ExecutePlanAction}}, the reality is that it still can take significant
amount of time until the moment when the new replicas fully recover and cause the switch of
shard states (parent to INACTIVE, child from RECOVERY to ACTIVE).

If this time is longer than the trigger's {{waitFor}} the trigger will issue the same SPLITSHARD
request again. {{SplitShardCmd}} doesn't prevent this new request from being processed because
the parent shard is still ACTIVE. However, a section of the code in {{SplitShardCmd}} will
realize that sub-slices with the target names already exist and they are not active, at which
point it will delete the new sub-slices ({{SplitShardCmd:182}}).

The end result is an infinite loop, where {{IndexSizeTrigger}} will keep generating SPLITSHARD,
and {{SplitShardCmd}} will keep deleting the recovering sub-slices created by the previous

A simple solution is for the parent shard to be marked to indicate that it’s in a process
of splitting, so that no other split is attempted on the same shard. Furthermore, {{IndexSizeTrigger}}
could temporarily exclude such shards from monitoring.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message