kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian McKee <florian.mc...@gmail.com>
Subject JDBC Sink Connector: Counterpart to Dead Letter Queue to keep track of successfully processed records
Date Tue, 27 Apr 2021 21:18:39 GMT
Hi,


I want to ingest messages that are sent by 3rd parties into our system.


The ingestion process is as follows:

   - verify message content
   - forward invalid messages to ingestion_failure topic
   - persist valid messages in PostgreSQL
   - forward messages that have been persisted to ingestion_success topic


The last point is key: Only forward messages to ingestion_successful if
they have been persisted in PostgreSQL. Is there any way I can do that with
a JDBC sink connector?

I'm basically looking for the counterpart of the Dead Letter Queue for
records that have been processed successfully by the connector.

If that's beyond the scope of connectors: Any recommendations on how to do
that any other way?



Thanks
Florian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message