kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Javier Holguera <javier.holgu...@gmail.com>
Subject Leveraging DLQ for errors coming from a sink connector plugin
Date Tue, 12 Nov 2019 17:40:15 GMT
Hi,

Looking at the Kafka Connect code, it seems that the built-in support for
DLQ queues only works for errors related to transformations and converters
(headers, key, and value).

I wonder if it has been considered (and maybe discarded) to use the same
mechanism for the call to the connector-plugin.put() operation. That way,
it would be possible for connector-plugins to leverage the same DLQ
semantics that Connect already implements without "reinventing the wheel"
themselves.

We have found ourselves in that situation with one in-house connector
plugin that we are building (HTTP Connector with extra specific bits). When
the connector plugin has tried X times to do its HTTP call, we want to
"offload" the record into a DLQ queue.

Any chances that this will be implemented?

Thanks.

Regards,
Javier.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message