kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bin Wang <binwang...@gmail.com>
Subject Pulling data from remote server
Date Tue, 29 Dec 2015 22:26:05 GMT
Hi there,

I have several different production servers running in different cloud
providers, like AWS, Rackspace, ..etc. And I have one central big data
environment inside company's network. I need to build a NRT(near real time)
monitoring system which I need to stream and archive all the logs from all
those different servers into the central place.

I have read through the documentation of Kafka and found that it seems
like Kafka
Connect <http://kafka.apache.org/documentation.html#quickstart_kafkaconnect>
does what I want. However, instead of connecting to a local file, I really
need to connect to a file that is on a remote server. Even more
challenging, due to the existence of the firewall, I can only pull from the
remote server into the big data environment instead of pushing from the
remote to the big data environment.

Is there any built in functionality in Kafka does what I want? If not, what
is the best practice to architect this?

I have seen how Splunk and Sumologic and really amazed at how well they
works, however, at least Sumologic requires to put a connector on the
remote server and it will periodically check and update the logs to sumo's
server. It is kind of cool but really won't work in my scenario due to the

I have asked something similar on Stackoverflow
in case you are interested in getting a few points there :)

Best regards,


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message