kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Smyth <smyth.james...@gmail.com>
Subject Re: what are common ways to convert info on a web site into a log entry?
Date Mon, 15 Jan 2018 20:23:34 GMT
thanks!  I will write my reply into the mail group once I have fiddled with your suggestions.


> On Jan 15, 2018, at 2:20 PM, Jacob Sheck <shec0002@gmail.com> wrote:
> You will need to find or create something to accomplish this.  Topics in
> Kafka primarily act as queues.  If you search the web for more information
> about "kafka connector http" you will find a few projects that do this.
> You could also take a look at the Confluent S3 Connector as an example of
> how to accomplish what you want to do.  If you are interested in a quick
> and dirty solution you could put a script using curl in cron to pull the
> web content and write it to a topic using the kafka-rest-proxy.
> On Mon, Jan 15, 2018 at 1:52 PM James Smyth <smyth.james965@gmail.com>
> wrote:
>> Hi Kafka people,
>> I am very new to kafka, so perhaps my question is naive. I spent some time
>> searching around at resources of kafka but only became more confused.
>> What are common ways to pull info from a web site and send to kafka to
>> become a log entry?
>> There is a web site that I want to pull a piece of data from once/month
>> and have that data written to Kafka log. Consumers will be listening for
>> that message to do processing on it.
>> I am not sure about common ways to do this.
>> I am thinking I could have some scheduler (e.g. cron) wake up once per
>> month and trigger the pull of the data from the web site and then send it
>> to a kafka stream.
>> Does kafka have ability to trigger events once/month or is using cron a
>> better idea?
>> What is the scheduler triggering a stand-alone batch job or the running of
>> a some service like a kafka producer? Should I worry about a service
>> running all the time when it is likely to only do a few seconds of work
>> each month?
>> Many thanks,
>> James.

View raw message