kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hans Jespersen <h...@confluent.io>
Subject Re: is there a way to make sure two consumers receive the same message from the broker?
Date Mon, 07 Nov 2016 00:25:02 GMT
That depends on what system you are using for storage. Are you talking about databases, file
systems, or perhaps some other form of storage? 
What is the rate and size and format of the updates? Are the two storage systems in the same
datacenter or separated by a WAN? 
What programming language and operating system are you using to build this system? You have
to give us more information if you want specific recommendations.

-hans

> On Nov 6, 2016, at 2:54 PM, kant kodali <kanth909@gmail.com> wrote:
> 
> Hi! Thanks. any pointers on how to do that?
> 
> On Sun, Nov 6, 2016 at 2:32 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com>
> wrote:
> 
>> You should have one consumer pull the message and submit the data to each
>> storage using an XA transaction.
>> 
>>> On Nov 5, 2016, at 19:49, kant kodali <kanth909@gmail.com> wrote:
>>> 
>>> yes this problem can definetly be approached in many ways but given the
>>> hard constraints by our clients we don't seem to have many options. so
>> the
>>> problem is we have to keep two storages systems in sync all the time. so
>>> whatever the data that is storage 1 should also be in storage 2 at any
>>> given time. so we explored the following options
>>> 
>>> 1)  we thought about ETL from storage 1 to storage 2 but that approach
>> has
>>> bunch of  drawbacks given the time constraints.
>>> 2)  Add a common service on top of two storages and do some sort of 2PC
>> but
>>> that would degrade the write performance. Morever we dont really have a
>>> control over how fast each write/store can happen at each storage layer
>>> (because these two storages are completely different).
>>> 
>>> so I started exploring if there is any tricks I could do with Kafka?
>>> 
>>> 
>>> 
>>>> On Sat, Nov 5, 2016 at 5:01 PM, Hans Jespersen <hans@confluent.io>
>> wrote:
>>>> 
>>>> Yes exactly. If consumer 1 gets message with offset 17 then it can write
>>>> that offset into an external storage that consumer 2 can also check to
>>>> ensure that it keeps in sync with consumer 1.
>>>> 
>>>> Just curious though why you would need to do this? What is the use case
>>>> because there may be a better way to get you the functionality you want?
>>>> 
>>>> -hans
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Nov 5, 2016, at 4:31 PM, kant kodali <kanth909@gmail.com> wrote:
>>>>> 
>>>>> I am new to Kafka and reading this statement "write consumer 1 and
>>>> consumer
>>>>> 2 to share a common external offset storage" I can interpret it many
>> ways
>>>>> but my best guess is as follows.
>>>>> 
>>>>> Are you saying write the current offset of each consumer to a common
>>>>> external storage?
>>>>> 
>>>>> 
>>>>>> On Sat, Nov 5, 2016 at 4:15 PM, kant kodali <kanth909@gmail.com>
>> wrote:
>>>>>> 
>>>>>> Hi Hans,
>>>>>> 
>>>>>> What do you mean by "write consumer 1 and consumer 2 to share a common
>>>>>> external offset storage" ? can you please elaborate a bit more.
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> On Sat, Nov 5, 2016 at 4:00 PM, Hans Jespersen <hans@confluent.io>
>>>> wrote:
>>>>>> 
>>>>>>> There is no built in mechanism to do this in Apache Kafka but
if you
>>>> can
>>>>>>> write consumer 1 and consumer 2 to share a common external offset
>>>> storage
>>>>>>> then you may be able to build the functionality you seek.
>>>>>>> 
>>>>>>> -hans
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Nov 5, 2016, at 3:55 PM, kant kodali <kanth909@gmail.com>
wrote:
>>>>>>>> 
>>>>>>>> Sorry there is a typo. here is a restatement.
>>>>>>>> 
>>>>>>>> Is there a way to make sure two consumers receive the same
message
>>>> from
>>>>>>> the
>>>>>>>> kafka broker in a atomic way? such that if consumer 1 gets
a message
>>>>>>>> consumer 2 should also get that message and if  consumer
1 fails for
>>>>>>>> whatever reason consumer 2 should also rollback to previous
offset
>> (to
>>>>>>> the
>>>>>>>> same offset as consumer 1) or invalidate or something like
that. is
>>>> that
>>>>>>>> possible?
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> This e-mail and any files transmitted with it are confidential, may
>> contain sensitive information, and are intended solely for the use of the
>> individual or entity to whom they are addressed. If you have received this
>> e-mail in error, please notify the sender by reply e-mail immediately and
>> destroy all copies of the e-mail and any attachments.
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message