kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiangjie Qin <j...@linkedin.com.INVALID>
Subject Re: Get last snapshot from compacted topic
Date Sat, 08 Aug 2015 17:21:29 GMT
You are looking for a K-V store here, so the general answer is no. But
Kafka does have an internal K-V store but only for consumer offsets. So
there are some tricks we can play:

If your processing node is consuming from Kafka and the offset of a
snapshot is the offset of a Kafka partition, you can simply commit the
offsets to Kafka. Later on when the node restarts, you can fetch the
committed offset and that will be what you want.

If you snapshot offsets has nothing to do with Kafka, you may do the
following:
1. produce your snapshot to the snapshot checkpoint topic and get back the
offset of the produced message.
2. use a consumer to commit the offset returned by broker in step 1 using
the node name as the group name for the checkpoint topic partition.
3. When restart, fetch the offset of using node name as group name and that
offset is what you need.

Thanks,

Jiangjie (Becket) Qin

On Tue, Aug 4, 2015 at 3:16 AM, Aki <jack-akfak@marmelandia.com> wrote:

> I'd like to save a snapshot of a processing node's state in a compacted
> kafka topic. A large number of nodes would save their snapshots in the same
> partition.
>
> What is an efficient way for a (restarted) node to find the offset of its
> latest snapshot? Using just Kafka (no database, local file, etc.), is there
> a more efficient way than to consume the partition from the earliest
> available offset (potentially reading a lot of snapshots of other nodes).
>
> Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message