kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From noah <iamn...@gmail.com>
Subject Re: Using Kafka as a persistent store
Date Fri, 10 Jul 2015 13:00:43 GMT
I don't want to endorse this use of Kafka, but assuming you can give your
message unique identifiers, I believe using log compaction will keep all
unique messages forever. You can read about how consumer offsets stored in
Kafka are managed using a compacted topic here:
http://kafka.apache.org/documentation.html#distributionimpl  In that case,
the consumer group id+topic+partition forms a unique message id and the
brokers read that topic on startup into the offsets cache (and take updates
to the offsets cache via the same topic.) If you have a finite, smallish
data set that you want indexed in multiple systems, that might be a good

If your data can grow without bound, it doesn't seem to me like Kafka is a
good choice? Even with compaction, you will still have to sequentially read
it all, message by message, to get it into a different system. As far as I
know, there is no lookup by id, and even going to a specific date is a
manual O(log n) process.

(warning: I'm just another user, so I may have a few things wrong.)

On Fri, Jul 10, 2015 at 3:47 AM Daniel Schierbeck <
daniel.schierbeck@gmail.com> wrote:

> I'd like to use Kafka as a persistent store – sort of as an alternative to
> HDFS. The idea is that I'd load the data into various other systems in
> order to solve specific needs such as full-text search, analytics, indexing
> by various attributes, etc. I'd like to keep a single source of truth,
> however.
> I'm struggling a bit to understand how I can configure a topic to retain
> messages indefinitely. I want to make sure that my data isn't deleted. Is
> there a guide to configuring Kafka like this?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message