kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Wilson <dan...@mojotech.com>
Subject Connect offsets not persisting when worker is restarted
Date Mon, 22 Oct 2018 20:24:51 GMT
Are connector offsets persisted across worker restarts? I had thought they
would be since they are written to the connect-offsets topic. I have code
set up to save the offset as a field in my SourceTask, which works fine. I
can restart the task and the offsets seem to be persisted, though I don't
know where. Does the task stay in memory when restarted and isn't
re-instantiated? If so then it would just be reading the offsets from the
object itself.

I've checked and the connect-offsets topic has data written to it that
seems to match what I'd expect.

The problem comes when I restart the worker itself. It seems to start right
at the beginning again. I'm using the OffsetStorageReader on context to get
the offset at start and at poll.

This is a cluster of 1 kafka broker, 1 zookeeper node, and 1 kafka connect
worker running in distributed mode. Everything is running on the same VM.

Any hints or tips? Anything else I should look into?

Thanks,
Daniel Wilson-Thomas

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message