kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Smyth <smyth.james...@gmail.com>
Subject what are common ways to convert info on a web site into a log entry?
Date Mon, 15 Jan 2018 19:47:34 GMT
Hi Kafka people,

I am very new to kafka, so perhaps my question is naive. I spent some time searching around
at resources of kafka but only became more confused.

What are common ways to pull info from a web site and send to kafka to become a log entry?
There is a web site that I want to pull a piece of data from once/month and have that data
written to Kafka log. Consumers will be listening for that message to do processing on it.
I am not sure about common ways to do this.

I am thinking I could have some scheduler (e.g. cron) wake up once per month and trigger the
pull of the data from the web site and then send it to a kafka stream.
Does kafka have ability to trigger events once/month or is using cron a better idea?
What is the scheduler triggering a stand-alone batch job or the running of a some service
like a kafka producer? Should I worry about a service running all the time when it is likely
to only do a few seconds of work each month?

Many thanks,

View raw message