kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean Bic <jean.b...@gmail.com>
Subject Re: Kafka versus classic central HTTP(s) services for logs transmission
Date Sun, 21 Oct 2012 08:44:56 GMT
Joe:

Thanks for you answer, but we're trying to push Kafka Broker at each site...
... so your answer makes me realize why we're trying to push Kafka over
per-producers services call: that would make a very large number of
services call from each site (our logs producers gather data every 5
minutes, on average 100 items of about 128 bytes per machines, and we're
targeting from 250 to 4000 machines per "site").

I think that, with these numbers, we have a way make IT people understand
that Kafka solution will avoid flooding the site's firewall infrastructure
(which is active for outbound connections).
Beyond this good point for Kafka in terms of # of concurrent connections, I
am wondering if we could find other assets for Kafka solution...

Jean

-----Original Message-----
From: Joe Stein [mailto:cryptcom@gmail.com]
Sent: Sunday, October 21, 2012 1:26 AM
To: kafka-users@incubator.apache.org
Subject: Re: Kafka versus classic central HTTP(s) services for logs
transmission

You could move the producer code to the "site" and expose that as a REST
interface.

You can then benefit from the scale and consumer functionality that comes
with Kafka without these issues you are bringing up.

On Oct 20, 2012, at 4:27 PM, Jean Bic <jean.bicc@gmail.com> wrote:

> Hello,
>
> We have started to build a solution to gather logs from many machines
> located in various “sites” to a so-called “Consolidation server” which
role
> is to persists the logs and generate alerts based on some criteria
> (patterns in logs, triggers on some values, etc).
>
>
> We are challenged by our future users to clarify why Kafka is for this
need
> the best possible communication solution. They argue that it would be
> better to choose a more classic HTTP(S) based solution with producers
> calling REST services on a pool of Node.js servers behind a load-balancer.
>
>
> One of the main issue they see with Kafka is that  It requires connections
> from Consolidation Server to Kafka brokers and to Zookeeper daemons
located
> in each “site”, versus connections from logs producers in all sites to the
> Consolidation servers.
> Here Kafka is seen as a burden for each site’s IT team requiring some
> firewall special setup, versus. no firewall setup with the service-based
> solution :
>
> 1.      Kafka requires for each site IT team to create firewall rules for
> accepting incoming connections for a “non standard” protocol from the
> “Collector server” site
>
> 2.      IT team must expose all Zookeeper and Broker machines/ports to the
> “Collector server” site
>
> 3.   Kafka has no built-in encryption for data, where as a classic
services
> oriented solution can rely on HTTPS (reverse) proxies
>
> 4.      Kafka is not commonly known by IT people who do not know how to
> scale it: when should they add broker machines versus when should they add
> zookeeper machines?
>
> With the services-based solution, the IT teams of each site are free of
> scalability issues, only on “Consolidation server” site one has to add
> Node.js machine to scale up.
>
> I agree that these IT concerns can't be taken lightly.
>
> I need help from Kafka community to find rock solid assets for using Kafka
> over classic services-based solution.
>
> How would you “defend” Kafka against above “attacks”?
>
>
> Regards,
>
> Jean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message