kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tauzell, Dave" <Dave.Tauz...@surescripts.com>
Subject RE: Time based data retrieval from Kafka Topic
Date Tue, 05 Sep 2017 14:28:43 GMT
What are you going to do with the messages every 15 minutes?

One way I can think of is to have two consumers in your application.  One of them reads messages
and just keeps track of the offsets for each hour.  The other consumer then uses this info
to pull the data.  You could publish these offsets to a topic keyed on something like yyyy-mm-dd-hh24
or keep them in memory ( if running in the same application ).   You would need some one-time
process to create the offsets for the first time.

-Dave

-----Original Message-----
From: Kaustuv Bhattacharya [mailto:kaustuvlive@gmail.com]
Sent: Sunday, September 3, 2017 1:08 PM
To: users@kafka.apache.org
Subject: Time based data retrieval from Kafka Topic

*Hi All,*

*I need a solution to the below Problem statement -*

*How to retrieve only last 1 hour data from an existing Kafka Topic, on 1st & every consecutive
(at every 15 mins interval) of the client application?*
*Note:- The existing Topic is accumulating data since last 6 months.*

*Regards,*
*Kaustuv Bhattacharya*
This e-mail and any files transmitted with it are confidential, may contain sensitive information,
and are intended solely for the use of the individual or entity to whom they are addressed.
If you have received this e-mail in error, please notify the sender by reply e-mail immediately
and destroy all copies of the e-mail and any attachments.
Mime
View raw message