kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niek Sanders <niek.sand...@gmail.com>
Subject Re: Embedding a broker into a producer?
Date Thu, 12 Apr 2012 21:38:15 GMT
> What do you see as the downside to running a KAFKA Broker and letting
> it write your files locally?

Here is every downside I can come up with.  Some are trivial, but I've
tried to play devil's advocate.

1) Additional memory/processing footprint, as you mentioned.

2) Additional network usage by Kafka consumers hitting producer box.

3) Loss of either producer elasticity or message retention.  One of
the awesome features of Kafka is being able to hold on a long history
of messages and replay as needed.  But if the producers machines hold
my brokers, I can no longer scale down the number of producer machines
as the system load drops--a loss of elasticity.  This ties to another
group discussion about decommissioning brokers.

4) Security... some producers live on web-accessible machines.  The
less open ports accepting incoming connections, the better.  Not a
huge issue with good firewall rules, but still something to ponder.

5) Loss of data duplication has downsides too.  Having multiple copies
of data does violate DRY, but it can also add robustness.  If the
hard-disk on either the producer or the brokers dies, you still have
the data lying around.  Since the data is not getting changed on
either the broker or any producer log files, you shouldn't have
syncing issues normally associated with DRY violations.

- Niek

View raw message