kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AmirHossein Roozbahany <Band...@outlook.com>
Subject RE: is there a way to make sure two consumers receive the same message from the broker?
Date Tue, 08 Nov 2016 09:37:06 GMT
Excuse me this part was non-sense: if the latest update to a document in es always win in Cassandra's
LWW, they will "eventually" "converge".
________________________________
From: AmirHossein Roozbahany<mailto:BandBaz@outlook.com>
Sent: ‎11/‎8/‎2016 8:16 AM
To: users@kafka.apache.org<mailto:users@kafka.apache.org>
Subject: RE: is there a way to make sure two consumers receive the same message from the broker?

Generally Cassandra itself is not consistent enough, even with quorum read-writes, say one
of the writes fail, the nodes who received the data won't roll back and it might lead to dirty
reads which in turn makes roll back logic tricky (if not impossible). You can use linearizable
writes but if you want to use them all the time, why bother using Cassandra!?

The important thing about Cassandra is that all of the nodes will eventually have the same
data(after anti-entropy or read-repair). they are "convergent", they will "eventually" converge,
and practically they converge pretty fast.

I think what you might actually need is to make two databases convergent(not necessarily fully
consistent at any given time) if the latest update to a document in es leads always win when
Cassandra is doing es they will "eventually" "converge".

Doing so is easy, as fast as I know es assigns a _version number to a document and increases
it on every update, now if your use of in Cassandra insert statement as the "writetime". Now
when Cassandra is doing read repair the record with higher writetime will win.

Using es's document _version field is just one option, you can use something from you domain
or kafka's offset or machine timestamp (not recommended at all).

I hope it could help
________________________________
From: kant kodali<mailto:kanth909@gmail.com>
Sent: ‎11/‎7/‎2016 8:18 PM
To: users@kafka.apache.org<mailto:users@kafka.apache.org>
Subject: Re: is there a way to make sure two consumers receive the same message from the broker?

Hi AmitHossein,

I still don't see how that guarantees consistency at any given time. other
words how do I know at time X the data in Cassandra and ES are the same.

Thanks


On Mon, Nov 7, 2016 at 3:26 AM, AmirHossein Roozbahany <divergnt@outlook.com
> wrote:

> Hi
>
> Can you use elasticsearch _version field as cassandra's
> writetime?(_version is strictly increasing, cassandra uses writetime for
> applying LWW, so last write in elasticsearch will always win)
>
> It needs no transaction and makes databases convergent.
>
>
> ________________________________
> From: kant kodali <kanth909@gmail.com>
> Sent: Monday, November 7, 2016 3:08 AM
> To: users@kafka.apache.org
> Subject: Re: is there a way to make sure two consumers receive the same
> message from the broker?
>
> Hi Hans,
>
> The two storages we use are Cassandra and Elastic search and they are on
> the same datacenter for now.
> The Programming Language we use is Java and OS would be Ubuntu or CentOS.
> We get messages in JSON format so we insert into Elastic Search directly
> and for Cassandra we transform JSON message into appropriate model so we
> could insert into a Cassandra table.
> The rate we currently get is about 100K/sec which is awesome but I am
> pretty sure this will go down once when we implement 2PC or transactional
> writes.
>
> Thanks,
> kant
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message