kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: is there a way to make sure two consumers receive the same message from the broker?
Date Mon, 07 Nov 2016 17:36:50 GMT
Fixing typo's

Hi Tauzell,

Yeah our users want to query, do aggregations on Elastic Search directly
and we cannot have inconsistent data  because say the writes didn't make it
into Cassandra but made it to Elastic search then a simple aggregations
like count will lead to a wrong answer but again as @Hans pointed out this
is no longer a Kafka question and also your solution has merits in its own
way which I really appreciate it! your solution does make writes faster and
probably some performance penalty on the read side given repairs happen
during the read stage in Cassandra (We could check in both but since our
users query elastic search directly there is no way for us to check it in
Cassandra else we could go with your solution as well).

Basically, we use ES as an index for Cassandra since secondary indexes in
Cassandra (including the latest implementation SASI) doesn't work with our
use case since we have high cardinality columns (which means every row in a
column is unique so index on a high cardinality column is not very
efficient given the underlying data structure used by SASI, but with
inverted index which is used by ES is much faster).

We do use Apache Spark along with Cassandra and I am trying to explore
Succint http://succinct.cs.berkeley.edu/wp/wordpress/ and if everything
works out with Succint we can get rid of elastic search. The only thing
that I worry and still testing with Spark, Cassandra and Succint is whether
If the aggregations/computations of a column or search on particular
Cassandra field/column  can happen in real time given a big dataset (with
ES it does so the goal is to see if we can get somewhere close or perform
even better).

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message