kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Tamai <daniel.ta...@gmail.com>
Subject Re: Using Kafka as a persistent store
Date Tue, 14 Jul 2015 00:57:39 GMT
Using -1 for log.retention.ms should work only for 0.8.3 (
https://issues.apache.org/jira/browse/KAFKA-1990).

2015-07-13 17:08 GMT-03:00 Shayne S <shaynest113@gmail.com>:

> Did this work for you? I set the topic settings to retention.ms=-1 and
> retention.bytes=-1 and it looks like it is deleting segments immediately.
>
> On Sun, Jul 12, 2015 at 8:02 AM, Daniel Schierbeck <
> daniel.schierbeck@gmail.com> wrote:
>
> >
> > > On 10. jul. 2015, at 23.03, Jay Kreps <jay@confluent.io> wrote:
> > >
> > > If I recall correctly, setting log.retention.ms and
> log.retention.bytes
> > to
> > > -1 disables both.
> >
> > Thanks!
> >
> > >
> > > On Fri, Jul 10, 2015 at 1:55 PM, Daniel Schierbeck <
> > > daniel.schierbeck@gmail.com> wrote:
> > >
> > >>
> > >>> On 10. jul. 2015, at 15.16, Shayne S <shaynest113@gmail.com>
wrote:
> > >>>
> > >>> There are two ways you can configure your topics, log compaction and
> > with
> > >>> no cleaning. The choice depends on your use case. Are the records
> > >> uniquely
> > >>> identifiable and will they receive updates? Then log compaction is
> the
> > >> way
> > >>> to go. If they are truly read only, you can go without log
> compaction.
> > >>
> > >> I'd rather be free to use the key for partitioning, and the records
> are
> > >> immutable — they're event records — so disabling compaction altogether
> > >> would be preferable. How is that accomplished?
> > >>>
> > >>> We have a small processes which consume a topic and perform upserts
> to
> > >> our
> > >>> various database engines. It's easy to change how it all works and
> > simply
> > >>> consume the single source of truth again.
> > >>>
> > >>> I've written a bit about log compaction here:
> > >>>
> > http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/
> > >>>
> > >>> On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck <
> > >>> daniel.schierbeck@gmail.com> wrote:
> > >>>
> > >>>> I'd like to use Kafka as a persistent store – sort of as an
> > alternative
> > >> to
> > >>>> HDFS. The idea is that I'd load the data into various other systems
> in
> > >>>> order to solve specific needs such as full-text search, analytics,
> > >> indexing
> > >>>> by various attributes, etc. I'd like to keep a single source of
> truth,
> > >>>> however.
> > >>>>
> > >>>> I'm struggling a bit to understand how I can configure a topic
to
> > retain
> > >>>> messages indefinitely. I want to make sure that my data isn't
> deleted.
> > >> Is
> > >>>> there a guide to configuring Kafka like this?
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message