kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Kafka Streams: Read from one Bootstrap server & write to other
Date Wed, 02 Dec 2020 23:49:45 GMT
Eric,

this idea came up multiple time already, but it is much more difficult
to build than it seems, especially for supporting "exactly once". Kafka
transaction only work on a single cluster...

Maybe we will support it at some point, but the "workaround" to either
replicate the input or output topic(s) is fairly simple and thus it
seems not to be a "killer feature" but more like a "nice to have" (just
my personal opinion).


-Matthias


On 12/1/20 2:08 PM, Malcolm McFarland wrote:
> How about a lightweight MirrorMaker instance for just this topic?
> 
> Malcolm McFarland
> Cavulus
> 
> 
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
> 
> 
> On Tue, Dec 1, 2020 at 1:18 PM Eric Beabes <mailinglists19@gmail.com> wrote:
> 
>> Hmm.. Ok. Maybe this could be a feature request for a future release. This
>> can be accomplished easily in Spark Structured Streaming... just saying :)
>>
>> In Spark Structured Streaming we can have separate configurations for
>> 'readStream' & 'writeStream'. I am a bit surprised this is not available in
>> Kafka Streams.
>>
>> Even during development sometimes we want to read from a topic in a QA
>> cluster & write it to a topic on the Local cluster.
>>
>>
>>
>> On Tue, Dec 1, 2020 at 11:33 AM Matthias J. Sax <mjsax@apache.org> wrote:
>>
>>> KafkaStreams can only connect to a single cluster.
>>>
>>> If you really need to read from one cluster and write to another, you
>>> have 3 main options:
>>>
>>>  - use KafkaStreams on the source cluster and mirror the output topic
>>> from the source to the target cluster
>>>
>>>  - mirror the input topic from the source cluster to the target cluster
>>> and use KafkaStreams on the target cluster
>>>
>>>  - don't use KafkaStreams but plain consumer/producer
>>>
>>>
>>>
>>> -Matthias
>>>
>>> On 12/1/20 10:58 AM, Eric Beabes wrote:
>>>> I need to read from a topic in one bootstrap server & write it to
>> another
>>>> topic in another bootstrap server. Since there's only one
>>>> StreamsConfig.BOOTSTRAP_SERVERS_CONFIG property, I am wondering how to
>>>> accomplish this?
>>>>
>>>> Do I need to create 2 different KafkaStreams objects? One for reading &
>>> the
>>>> other for writing?
>>>>
>>>
>>
> 

Mime
View raw message