kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 김세곤 <sgkim...@naver.com>
Subject hello, I'm user in kafka having a question about 'commit'
Date Wed, 29 Jan 2014 06:51:55 GMT
 
hello, I'm user in kafka
 
I want find commit method from consumer to zookeeper, but can't find it
I want to make structure like that,
 
hadoop-consumer: 1. it get message
2. it write message to hadoop hdfs
3. it have to get message again from what it read ,when it is fault and recover.
( for example,  consumer was reading message
suddenly it was fault. during fault time, message is producing.
after recovering, consumer have to read message from what it didn't read offset, not from
first)
 
 
problem is 3.
 
I read about offset, written that
after consumer read message, it commit to zookeeper. using that offset ( ex, getbeforeoffset()
), can read message again.
But, using this method offer that reading from first(offset-1) to last. And if use not it,
consumer can't get messages produced during fault
 
So, I want find any source or method to satisfy my structure, is there any thing to help me?
 
 
thank you for reading
please give some help 
 
 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message