kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Hammerman <jhammer...@squarespace.com.INVALID>
Subject Resetting connect-offsets postion Using Kafka Connect and Debezium
Date Fri, 02 Mar 2018 20:56:19 GMT
HiApache Kafka users email distribution list.

I'm trying to post to connect-offsets topic a message with a lsn from the
past. I dump the connect offsets topic with the following command:

./kafka-console-consumer.sh --bootstrap-server <conn string>
--consumer.config ../config/consumer.properties --property print.key=true
--new-consumer --topic connect-offsets


(config-consumer.properties has a group.id and SSL configuration
information).

Then I'll extract a message, and keeping all the key & value fields the
same with the exception of using an old LSN, I turn off Kafka Connect and
produce to the connect-offsets topic with a connection formed with the
following command:

[root@billing-kafka001 bin]# ./kafka-console-producer.sh --broker-list   <conn
string> --topic connect-offsets --property "key.separator=," --property
"parse.key=true" --property "compression.codec=1" --producer.config
../config/producer.properties

(producer.properties has only ssl configuration settings).

The messages look like this:

["postgres-events-connector",{"server":"staging-billing"}]{"last_snapshot_record":true,"lsn":1275604312,"txId":2125,"ts_usec":1520019207386333000,"snapshot":true}

When Kafka Connect is restarted, however, the offset source_info reported
is that of the last commit issued by the producer, not my reset point.

Am I obtaining the key in the wrong fashion? Should txId be reset to the
value of commit I obtained the LSN from (I tested this and my results were
no better)? Am I using the correct compression codec for the topic?

Any assistance would be greatly appreciated!!

Thanks in advance for any assistance anyone can provide,
Joseph Hammerman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message