kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tauzell, Dave" <Dave.Tauz...@surescripts.com>
Subject RE: what are common ways to convert info on a web site into a log entry?
Date Tue, 16 Jan 2018 15:01:34 GMT
I would have a cron that runs every day but somehow tracks if it has pulled data for the month.
 If it has it just does nothing.  This way if you have some sort of failure one day (website
is down, etc ...) it would pull data the next day.

You could possibly use Kaka itself to store the last month that it grabbed data for.   Running
once a day is just an example, but the basic idea is to have some way of automatically dealing
with failures.    You might also want some way to monitor monthly in case it just stops working


-----Original Message-----
From: James Smyth [mailto:smyth.james965@gmail.com]
Sent: Monday, January 15, 2018 1:48 PM
To: users@kafka.apache.org
Subject: what are common ways to convert info on a web site into a log entry?

Hi Kafka people,

I am very new to kafka, so perhaps my question is naive. I spent some time searching around
at resources of kafka but only became more confused.

What are common ways to pull info from a web site and send to kafka to become a log entry?
There is a web site that I want to pull a piece of data from once/month and have that data
written to Kafka log. Consumers will be listening for that message to do processing on it.
I am not sure about common ways to do this.

I am thinking I could have some scheduler (e.g. cron) wake up once per month and trigger the
pull of the data from the web site and then send it to a kafka stream.
Does kafka have ability to trigger events once/month or is using cron a better idea?
What is the scheduler triggering a stand-alone batch job or the running of a some service
like a kafka producer? Should I worry about a service running all the time when it is likely
to only do a few seconds of work each month?

Many thanks,


This e-mail and any files transmitted with it are confidential, may contain sensitive information,
and are intended solely for the use of the individual or entity to whom they are addressed.
If you have received this e-mail in error, please notify the sender by reply e-mail immediately
and destroy all copies of the e-mail and any attachments.

View raw message