kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Stopford <...@confluent.io>
Subject Re: [!!Mass Mail]Re: Dual commit with zookeeper and kafka
Date Sat, 10 Oct 2015 10:35:17 GMT
Dual commits are only really designed to aid migration of topics that were started using ZK
and want to move offset storage to Kafka. They are completely separate offset storage mechanisms.
I suggest you pick one and use that. I’d suggest Kafka. 

In the crash case you describe, offsets would be different in ZK & Kafka. The consumer
would pick the larger of the two offsets to use.

> On 10 Oct 2015, at 11:11, Рябков Алексей Николаевич <a.ryabkov@ntc-vulkan.ru>
wrote:
> 
> Hello!
> 
>>> Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or
both depends on the settings in two properties: offset.storage and >>dual.commit.enabled
(here <http://kafka.apache.org/documentation.html#consumerconfigs>). Currently ZK is
the default.
> 
> Thanks for explanation.... but I have more questions...
> 1. As far as I understand when I use offset.storage=kafka  and dual.commit.enabled in
consumer (I mean java-based) then when I call commit method commit then internally client
commit offset  to kafka and then to zk.... Is it correct?
> 2. if first is correct ...then  what happened if client crash between this  2 operations
(commit 2 kafka and commit 2 zookeeper)?
> 3. What happened if broker crash.... And my client support only zookeeper? (I guess that
I read message from beggining) Is it correct?  Or broker explicitly sync kafka commit with
zookeeper when started?
> 
> Thanks in advance, Aleksey 
> 
> -----Original Message-----
> From: Ben Stopford [mailto:ben@confluent.io] 
> Sent: Friday, October 9, 2015 5:54 PM
> To: users@kafka.apache.org
> Subject: [!!Mass Mail]Re: Dual commit with zookeeper and kafka
> 
> Hi Alexey
> 
> Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both depends
on the settings in two properties: offset.storage and dual.commit.enabled (here <http://kafka.apache.org/documentation.html#consumerconfigs>).
Currently ZK is the default. 
> 
> Commits happen either periodically (auto.commit.enable=true && auto.commit.interval.ms)
or explicitly. For example the high level consumer has a commitOffsets method. I expect the
python bindings have a similar function. The rest proxy offers this function too. 
> 
> Hope that helps 
> 
> B
> 	
> 
>> On 6 Oct 2015, at 23:06, Рябков Алексей Николаевич <a.ryabkov@ntc-vulkan.ru>
wrote:
>> 
>> Hello!
>> 
>> Can somebody explain me how to  use multiple consumers with different commit storage...
>> For example, java-based consumers use kafka commit storage...
>> python-based consumers use zookeeper commit storage
>> 
>> My question is:
>> Is it true that when one consumer commit to kafka,  server also commit 
>> automatically to zookeeper (so all consumers see the same commit) Is 
>> it true that when one consumer commit to zookeeper,  server also 
>> commit automatically to kafka (so all consumers see the same commit) 
>> Is it true that after server restart it read commits from kafka 
>> storage and publish (sync) commit to zookeeper (so after server 
>> restart all consumers can see also the same commit)
>> 
>> Thanks in advance, Aleksey
> 


Mime
View raw message