kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From R S <mypostbo...@gmail.com>
Subject Can hadoop-consumer be time based instead of offset based
Date Fri, 13 Apr 2012 08:01:38 GMT
Hi ,

I looked at hadoop-consumer , which fetches data directly from the kafka
broker . But from what i understand it is based on min and max offset and
map task complete once they reach the maximum offset for a given topic .

In our use case we would not know about the max offset before hand. Instead
we want map to keep reading data from a min offset and roll over every 30
mins . At 30th min we would again generate the offsets which would be used
for the next run.

any suggestions would be helpful .


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message