kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Рябков Алексей Николаевич <a.ryab...@ntc-vulkan.ru>
Subject RE: [!!Mass Mail]Re: Dual commit with zookeeper and kafka
Date Sat, 10 Oct 2015 10:11:06 GMT
Hello!

>> Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both
depends on the settings in two properties: offset.storage and >>dual.commit.enabled
(here <http://kafka.apache.org/documentation.html#consumerconfigs>). Currently ZK is
the default.

Thanks for explanation.... but I have more questions...
1. As far as I understand when I use offset.storage=kafka  and dual.commit.enabled in consumer
(I mean java-based) then when I call commit method commit then internally client commit offset
 to kafka and then to zk.... Is it correct?
2. if first is correct ...then  what happened if client crash between this  2 operations (commit
2 kafka and commit 2 zookeeper)?
3. What happened if broker crash.... And my client support only zookeeper? (I guess that I
read message from beggining) Is it correct?  Or broker explicitly sync kafka commit with zookeeper
when started?

Thanks in advance, Aleksey 

-----Original Message-----
From: Ben Stopford [mailto:ben@confluent.io] 
Sent: Friday, October 9, 2015 5:54 PM
To: users@kafka.apache.org
Subject: [!!Mass Mail]Re: Dual commit with zookeeper and kafka

Hi Alexey

Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both depends
on the settings in two properties: offset.storage and dual.commit.enabled (here <http://kafka.apache.org/documentation.html#consumerconfigs>).
Currently ZK is the default. 

Commits happen either periodically (auto.commit.enable=true && auto.commit.interval.ms)
or explicitly. For example the high level consumer has a commitOffsets method. I expect the
python bindings have a similar function. The rest proxy offers this function too. 

Hope that helps 

B
	

> On 6 Oct 2015, at 23:06, Рябков Алексей Николаевич <a.ryabkov@ntc-vulkan.ru>
wrote:
> 
> Hello!
> 
> Can somebody explain me how to  use multiple consumers with different commit storage...
> For example, java-based consumers use kafka commit storage...
> python-based consumers use zookeeper commit storage
> 
> My question is:
> Is it true that when one consumer commit to kafka,  server also commit 
> automatically to zookeeper (so all consumers see the same commit) Is 
> it true that when one consumer commit to zookeeper,  server also 
> commit automatically to kafka (so all consumers see the same commit) 
> Is it true that after server restart it read commits from kafka 
> storage and publish (sync) commit to zookeeper (so after server 
> restart all consumers can see also the same commit)
> 
> Thanks in advance, Aleksey

Mime
View raw message