kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Schierbeck <daniel.schierb...@gmail.com>
Subject Re: Using Kafka as a persistent store
Date Fri, 10 Jul 2015 20:55:48 GMT

> On 10. jul. 2015, at 15.16, Shayne S <shaynest113@gmail.com> wrote:
> 
> There are two ways you can configure your topics, log compaction and with
> no cleaning. The choice depends on your use case. Are the records uniquely
> identifiable and will they receive updates? Then log compaction is the way
> to go. If they are truly read only, you can go without log compaction.

I'd rather be free to use the key for partitioning, and the records are immutable — they're
event records — so disabling compaction altogether would be preferable. How is that accomplished?
> 
> We have a small processes which consume a topic and perform upserts to our
> various database engines. It's easy to change how it all works and simply
> consume the single source of truth again.
> 
> I've written a bit about log compaction here:
> http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/
> 
> On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck <
> daniel.schierbeck@gmail.com> wrote:
> 
>> I'd like to use Kafka as a persistent store – sort of as an alternative to
>> HDFS. The idea is that I'd load the data into various other systems in
>> order to solve specific needs such as full-text search, analytics, indexing
>> by various attributes, etc. I'd like to keep a single source of truth,
>> however.
>> 
>> I'm struggling a bit to understand how I can configure a topic to retain
>> messages indefinitely. I want to make sure that my data isn't deleted. Is
>> there a guide to configuring Kafka like this?
>> 

Mime
View raw message