kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Moving data from one cluster to another with Kafka Streams
Date Fri, 02 Feb 2018 23:36:02 GMT
Kafka Streams only work with a single cluster.

Thus, you would need to either transform the data first and replicate
the output topic to the target cluster, or replicate first and transform
within the target cluster.

Note, for the "intermediate" topic you need, you can set a low retention
time to reduce storage footprint, as it only acts as a temporal topic
and the actual data is safely stored in the original source and final
target topic.

As an alternative, you might want to check out
"single-message-transforms" (SMT) using Kafka Connect. Those allow you
to do simple transformation on the fly while copying data around. If you
don't need advanced transformations like aggregations or joins, SMT
might be sufficient and you don't need to use Kafka Streams.


On 2/2/18 2:54 PM, Geoffrey Holmes wrote:
> I need to get messages from a topic in one Kafka cluster, transform the message payload,
and put the messages into topics in another Kafka cluster. Is it possible to do this with
Kafka Streams? I don’t see how I can configure the stream to use one cluster for the input
and another cluster for output.

View raw message