kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Garcia <dav...@spiceworks.com>
Subject Re: Time based data retrieval from Kafka Topic
Date Tue, 05 Sep 2017 14:31:41 GMT
Make a topic and then set the retention to 1 hour.  Every 15 minutes, start a consumer that
always reads from the beginning.


On 9/5/17, 9:28 AM, "Tauzell, Dave" <Dave.Tauzell@surescripts.com> wrote:

    What are you going to do with the messages every 15 minutes?
    One way I can think of is to have two consumers in your application.  One of them reads
messages and just keeps track of the offsets for each hour.  The other consumer then uses
this info to pull the data.  You could publish these offsets to a topic keyed on something
like yyyy-mm-dd-hh24 or keep them in memory ( if running in the same application ).   You
would need some one-time process to create the offsets for the first time.
    -----Original Message-----
    From: Kaustuv Bhattacharya [mailto:kaustuvlive@gmail.com]
    Sent: Sunday, September 3, 2017 1:08 PM
    To: users@kafka.apache.org
    Subject: Time based data retrieval from Kafka Topic
    *Hi All,*
    *I need a solution to the below Problem statement -*
    *How to retrieve only last 1 hour data from an existing Kafka Topic, on 1st & every
consecutive (at every 15 mins interval) of the client application?*
    *Note:- The existing Topic is accumulating data since last 6 months.*
    *Kaustuv Bhattacharya*
    This e-mail and any files transmitted with it are confidential, may contain sensitive
information, and are intended solely for the use of the individual or entity to whom they
are addressed. If you have received this e-mail in error, please notify the sender by reply
e-mail immediately and destroy all copies of the e-mail and any attachments.

View raw message