kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charan Ganga Phani Adabala <char...@eiqnetworks.com>
Subject Regarding The Kafka Offset Management Issue In Direct Stream Approach.
Date Mon, 02 Nov 2015 04:26:02 GMT
Hi All,
We are working in Apache spark with Kafka integration, in this use case we are using DirectStream
approach. we want to avoid the data loss in this approach for actually we take offsets and
saving that offset into MongoDB.
We want some clarification is Spark stores any offsets internally, let us explain some example
:
For the first rdd batch we get 0 to 5 offsets of events to be processed, but unexpectedly
the application is crashed, then we started aging the application, then this job fetches again
from 0 to 5 events or where the event stopped in previous job.
We are not committing any offsets in the above process, because we have to commit offsets
manually in DirectStream approach. Is that new job fetches events form 0th position.


Thanks & Regards,
Ganga Phani Charan Adabala | Software Engineer
o:  +91-40-23116680 | c:  +91-9491418099
e:  charana@eiqnetworks.com<mailto:charana@eiqnetworks.com>
[cid:image001.jpg@01CF60B1.87C0C870]
EiQ Networks(r), Inc. |  www.eiqnetworks.com<http://www.eiqnetworks.com/>
www.socvue.com<http://www.socvue.com/> | www.eiqfederal.com<http://www.eiqfederal.com/>

[Blog]<http://blog.eiqnetworks.com/>Blog<http://blog.eiqnetworks.com/>   [Twitter]
<https://twitter.com/eiqnetworks>  Twitter<https://twitter.com/eiqnetworks>  
[LinkedIn] <http://www.linkedin.com/company/eiqnetworks>  LinkedIn<http://www.linkedin.com/company/eiqnetworks>
  [Facebook] <http://www.facebook.com/eiqnetworks>  Facebook<http://www.facebook.com/eiqnetworks>

"This email is intended only for the use of the individual or entity named above and may contain
information that is confidential and privileged. If you are not the intended recipient, you
are hereby notified that any dissemination, distribution or copying of the email is strictly
prohibited. If you have received this email in error, please destroy the original message."



Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message