lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: If the leader dies, will the data be lost?
Date Tue, 07 Apr 2020 15:05:33 GMT
bq. Does the replication by tlog always work like this?

Yes. Otherwise there’s no way to even attempt to guarantee data integrity. The raw updates
_must_ be received by all non-leaders that are eligible (i.e. TLOG or NRT but not PULL replicas).

BTW, anecdotally I’ve seen indexing throughput take quite a hit because of this, like 30%
or more when comparing leader-only with leader + NRT or TLOG replicas. This was an outlier
because the records were very small and the communications overhead to forward the raw docs
and wait for an ack back was large in comparison to the time it actually took to index the
docs. Most installations have much less of a difference.

And, BTW, the docs are forwarded to individual replicas in parallel.

Best,
Erick

> On Apr 6, 2020, at 10:08 PM, Taisuke Miyazaki <miyazakitaisuke@lifull.com> wrote:
> 
> Hi Erick,
> 
>> Before the leader goes down, the sequence of an update is this.
>> - the doc comes in to the leader (TL)
>> - the doc is forwarded to all the other tlog replicas (TF) and written to
> _their_ tlogs
>> - all the TF replicas ack back to TL
>> - TL acks back to the client
> 
> So the write request doesn't return a response until all the tlog replicas
> have been written!
> Thank you.
> 
> BTW, does replication by tlog always work like this?
> 
> 
> 
> 
> 2020年4月6日(月) 20:42 Erick Erickson <erickerickson@gmail.com>:
> 
>> You’ve got the sequence, that’s it exactly.
>> 
>> I don’t quite understand the second part of the question, but let me
>> address data loss.
>> 
>> Before the leader goes down, the sequence of an update is this.
>> - the doc comes in to the leader (TL)
>> - the doc is forwarded to all the other tlog replicas (TF) and written to
>> _their_ tlogs
>> - all the TF replicas ack back to TL
>> - TL acks back to the client
>> 
>> So, upon getting success back from the update request, all TLOG replicas
>> have the
>> docs in their local tlog files. So when the leadership changes, the new
>> leader
>> has all the docs to replay, thus no data loss.
>> 
>> At that point, the old leader’s tlogs are irrelevant. When it  comes back
>> online,
>> the sequence is:
>> 
>> - synch from the new leader, including any tlogs. This effectively erases
>> the old tlogs
>> - start writing any new docs into the local tlog
>> 
>> The old leader then remains a follower until some event changes things
>> again.
>> 
>> Best,
>> Erick
>> 
>>> On Apr 6, 2020, at 1:53 AM, Taisuke Miyazaki <miyazakitaisuke@lifull.com>
>> wrote:
>>> 
>>> Hi,
>>> Using solr 7.5.0 on solr cloud, and replica type is tlog.
>>> 
>>> If a leader dies, how is the re-election of the leader and the
>>> synchronization of the replicas done?
>>> 
>>> In my opinion.
>>> Leader dies→ New tlog replica tries to become Leader→ Replays tlogs not
>>> reflected in the index→ Becomes Leader
>>> Is this the right fit first?
>>> 
>>> Also, when another leader is elected, does it create a tlog that is only
>>> available to the old leader? (I'm worried about data being lost if the
>>> tlogs aren't synchronized.)
>> 
>> 


Mime
View raw message