lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christine Poerschke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
Date Wed, 10 Jul 2019 19:58:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882405#comment-16882405
] 

Christine Poerschke commented on SOLR-13257:
--------------------------------------------

Thanks [~mgibney] for creating and working on this ticket!

I have some "requirements" or "intended behaviour" type questions or thoughts, perhaps best
posed via examples.

1. The out-of-the-box behaviour (in the absence of {{shards.preference}} or similar parameters)
is, as you say, a random choice.
 * 
 ** Example: A collection with three replicas (replicaA, replicaB, replicaC), all of which
are available at the time of searching.
 ** Expected behaviour: The list of available replicas is randomly shuffled i.e. there are
{{3! = 6}} possibilities overall: {{[ [A, B, C], [A, C, B], [B, A, C], [B, C, A], [C, A, B],
[C, B, A] ]}}

2. The objective of this ticket i.e. the proposed behaviour (in the absence of {{shards.preference}}
or similar parameters) is reduce or remove the random-ness of the out-of-the-box behaviour.
 * 
 ** Example: A collection with three replicas (replicaA, replicaB, replicaC), all of which
are available at the time of searching. The request indicates (directly via a numeric value
parameter or indirectly via a non-numeric value parameter that can be hashed) the affinity
to an element in or a portion of the list of replicas. Let's assume the affinity expressed
translates into replica {{A}}.
 ** Expected behaviour:
 *** Possibility 1: The preferred replica is {{A}} but beyond the first choice the random-ness
remains for load balancing purposes i.e. there are {{2! = 2}} possibilities overall: {{[ [A,
B, C], [A, C, B] ]}}
 *** Possibility 2: The preferred replica is {{A}} and also beyond that first choice there
is no random-ness i.e. the replica list is sorted deterministically to give just one possibility:
{{[A, B, C]}}

3. The existing {{shards.preference}} parameter influences the replica choices (in the absence
of new affinity logic).

(caveat: I presume the below is the existing behaviour of the existing code but have not recently
read the code in detail to fully convince myself that this is so.)
 * 
 ** Example 1: A collection with six replicas spread equally across three locations. The request
indicates a preference for "Europe" location.
 ** Expected behaviour: The preferred replicas (based on location) are at the start of the
list and the not-preferred replicas are at the end of the list. The ordering within the start
and end portions of the list is random giving {{2! * 4! = 2*1 * 4*3*2*1 = 2 * 24 = 48}} possibilities
overall.

 * 
 ** Example 2: A collection with six replicas spread equally across three locations and the
two replicas at each location are of different types e.g. {{PULL}} and {{TLOG}}. The request
indicates a preference primarily for "Europe" location and secondarily for "PULL" type.
 ** Expected behaviour: The preferred replicas (based on location) are at the start of the
list and the not-preferred replicas are at the end of the list. The ordering within the start
portion of the list becomes deterministic via the secondary preference (e.g. {{[ replicaB1(europe,pull),
replicaB2(europe,tlog) ]}}) and the ordering within the end portion of the list is narrowed
down via the secondary preference but some random-ness remains giving 4 possibilities overall:
{code:java}
[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaA1(america,pull), replicaC1(asia,pull),
  replicaA2(america,tlog), replicaC2(asia,tlog) ]

[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaA1(america,pull), replicaC1(asia,pull),
  replicaC2(asia,tlog), replicaA2(america,tlog) ]

[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaC1(asia,pull), replicaA1(america,pull),
  replicaC2(asia,tlog), replicaA2(america,tlog) ]

[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaC1(asia,pull), replicaA1(america,pull),
  replicaA2(america,tlog), replicaC2(asia,tlog) ]
{code}

4. The new shard affinity logic combined with a single {{shards.preference}} parameter.
 * 
 ** Example: A collection with six replicas spread equally across three locations. The request
indicates a preference for "Europe" location.
 ** Expected behaviour: The preferred replicas (based on location) are at the start of the
list and the not-preferred replicas are at the end of the list.
 *** The ordering within the "Europe" start portion of the list becomes deterministic via
the new shard affinity logic.
 *** Does the new shard affinity logic influence the ordering within the end portion of the
list too?

5. The new shard affinity logic combined with multiple {{shards.preference}} parameters.
 * 
 ** At its simplest each {{shards.preference}} parameter will have two choices e.g. "PULL"
vs. "TLOG" replica type or "Europe" vs. unknown replica location.
 ** Assuming two {{shards.preference}} parameters with two choices each the resulting list
will have four portions (each potentially containing multiple replicas):
{code:java}
primaryPreference=preferredValue, secondaryPreference=preferredValue

primaryPreference=preferredValue, secondaryPreference=notPreferredValue

primaryPreference=notPreferredValue, secondaryPreference=preferredValue

primaryPreference=notPreferredValue, secondaryPreference=notPreferredValue

{code}

 * 
 ** With more than two {{shards.preference}} parameters the number of portions in the list
will increase.
 ** Does the new shard affinity logic influence the ordering within first (start) portion
of the list only or does it influence the ordering within the other portions too?

----
End of brain dump, slightly longer than expected, oops.
----
Next re-reading the existing ticket comments and skimming over the {{.adoc}} changes in the
latest patch to try and answer the questions above:
 * Any {{shards.preference=replica.base:something}} parameter (other than {{shards.preference=replica.base:random}})
will remove all random-ness from the choices.

 * With respect to the new shard affinity logic's interaction with the existing {{shards.preference}}
parameters, in principle things could get quite complicated there but you very elegantly solve
that by designating that {{replica.base}} may be used only once and only as a tie-breaker
i.e. as the last of potentially multiple preferences.

 * Taking the {{shards.preference=replica.location:Europe,replica.type:PULL}} example from
the above, the addition of {{shards.preference=replica.base:stable:2:fooBar&fooBar=0}}
would result in the ordering ordering below because
 ** {{replicaA1(america,pull) and {{replicaC1(asia,pull)}} are equivalent (both don't match
the primary replica.location preference and both do match the secondary replica.type preference)
and {{replica.base:stable}} now determines their relative ordering
 ** {{replicaA2(america,tlog)}} and {{replicaC2(asia,tlog)}} are equivalent (both match neither
the primary replica.location preference nor the secondary replica.type preference) and {{replica.base:stable}}
now determines their relative ordering.
{code:java}
[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaA1(america,pull), replicaC1(asia,pull),
  replicaA2(america,tlog), replicaC2(asia,tlog) ]
{code}

Does my thought process above kind of make sense and are the resulting answers accurate/fair?

> Enable replica routing affinity for better cache usage
> ------------------------------------------------------
>
>                 Key: SOLR-13257
>                 URL: https://issues.apache.org/jira/browse/SOLR-13257
>             Project: Solr
>          Issue Type: New Feature
>          Components: SolrCloud
>            Reporter: Michael Gibney
>            Priority: Minor
>         Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch, SOLR-13257.patch
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request randomly
via [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
to a particular replica. In setups with replication factor >1, this normally results in
a situation where subsequent requests (which one would hope/expect to leverage cached results
from previous related requests) end up getting routed to a replica that hasn't seen any related
requests.
> The problem can be replicated by issuing a relatively expensive query (maybe containing
common terms?). The first request initializes the {{queryResultCache}} on the consulted replicas.
If replication factor >1 and there are a sufficient number of shards, subsequent requests
will likely be routed to at least one replica that _hasn't_ seen the query before. The replicas
with uninitialized caches become a bottleneck, and from the client's perspective, many subsequent
requests appear not to benefit from caching at all.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message