lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shalin Shekhar Mangar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-11239) Deprecate maxSHardsPerNode when autoscaling policies are used
Date Tue, 15 Aug 2017 12:59:01 GMT

    [ https://issues.apache.org/jira/browse/SOLR-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127179#comment-16127179
] 

Shalin Shekhar Mangar commented on SOLR-11239:
----------------------------------------------

To add some context behind this issue:

Firstly, {{maxShardsPerNode}} defaults to 1. So if anyone needs to host more than 1 replica
per node then they must specify {{maxShardsPerNode}} while creating the collection via the
collection API.

Secondly, when someone uses the bin/solr scripts to create a collection, the script automatically
calculates numShards*repFactor and passes that value as the maxShardsPerNode. This is a workaround
for the first point above. This effectively means that the script needs no maxShardsPerNode
limit at all.

Thirdly, the {{maxShardsPerNode}} is not stored in collection state and therefore it is only
applicable during the time of collection creation. Any future collection API invocations such
as add replicas or split shards do not respect {{maxShardsPerNode}}.

Thirdly, the policy framework has its own way of providing constraints similar to maxShardsPerNode
both globally (i.e. across all collections) and per-collection. These constraints are respected
by all collection APIs such as create, addreplica, splitshard, restore etc.

So maxShardsPerNode is really a half-broken solution to the problem of placing replicas on
nodes and we should deprecate it in favor of the capabilities exposed by the policy framework.
So how do we do this without surprising the user?

Noble is working on providing a patch that:
# Introduces {{maxShardsPerNode=-1}} as a way of saying that there is no limit and Solr should
just go ahead and create the collection as requested
# Change bin/solr to pass maxShardsPerNode=-1 by default.
# If a cluster policy exists or if a collection policy is specified then specifying maxShardsPerNode
will throw an exception during collection creation because otherwise it can conflict with
the limits specified in the policy. If users need a constraint like this then they must modify
the policy.

This way back compat is also preserved. The default maxShardsPerNode remains 1 as before and
users who don't need the policy framework can continue to rely on the existing behavior of
the maxShardsPerNode parameter.

> Deprecate maxSHardsPerNode when autoscaling policies are used
> -------------------------------------------------------------
>
>                 Key: SOLR-11239
>                 URL: https://issues.apache.org/jira/browse/SOLR-11239
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>    Affects Versions: 7.0
>            Reporter: Noble Paul
>            Assignee: Noble Paul
>            Priority: Blocker
>
> We have found out that {{maxShardPerNode}} is not compatible with the new auto scaling
policies. So we need to deprecate that parameter when the autoscaling policies are used.
> the {{bin/solr}} script passes that parameter all the time irrespective of whether the
user needs it or not. 
> We need to fix it for 7.0 itself



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message