kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian McKee <florian.mc...@gmail.com>
Subject Re: JDBC Sink Connector: Counterpart to Dead Letter Queue to keep track of successfully processed records
Date Tue, 27 Apr 2021 22:03:16 GMT
Interesting idea (I have to think about it).

On Tue, Apr 27, 2021 at 4:40 PM Ran Lupovich <ranlupovich@gmail.com> wrote:

> Just a thought , what about jdbc source connector from the database to the
> succsess topic?
>
> בתאריך יום ד׳, 28 באפר׳ 2021, 00:19, מאת Florian McKee ‏<
> florian.mckee@gmail.com>:
>
> > Hi,
> >
> >
> > I want to ingest messages that are sent by 3rd parties into our system.
> >
> >
> > The ingestion process is as follows:
> >
> >    - verify message content
> >    - forward invalid messages to ingestion_failure topic
> >    - persist valid messages in PostgreSQL
> >    - forward messages that have been persisted to ingestion_success topic
> >
> >
> > The last point is key: Only forward messages to ingestion_successful if
> > they have been persisted in PostgreSQL. Is there any way I can do that
> with
> > a JDBC sink connector?
> >
> > I'm basically looking for the counterpart of the Dead Letter Queue for
> > records that have been processed successfully by the connector.
> >
> > If that's beyond the scope of connectors: Any recommendations on how to
> do
> > that any other way?
> >
> >
> >
> > Thanks
> > Florian
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message