kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Can hadoop-consumer be time based instead of offset based
Date Fri, 13 Apr 2012 13:05:39 GMT
> we want map to keep reading data from a min offset and roll over every 30
> mins . At 30th min we would again generate the offsets which would be used
> for the next run.

Using the max offset would avoid deserializing the data. You could use
timestamp too, but for that you would need to include a timestamp in your
Kafka message and then deserialize data in the map task.

Thanks,
Neha

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message