kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Smith <esm...@stardotstar.org>
Subject Re: Embedding a broker into a producer?
Date Thu, 12 Apr 2012 21:53:09 GMT
Niek,

  Thanks for the response.  I agree with your assessments.

  It's a beautiful thing that I don't have to commit to this decision
early in the build process.  I can start it one way and just switch to
the other, as it is all a matter of configuration and not code.

  I'll let you know if we learn anything interesting along the way.

Ed

On Thu, Apr 12, 2012 at 5:38 PM, Niek Sanders <niek.sanders@gmail.com> wrote:
>> What do you see as the downside to running a KAFKA Broker and letting
>> it write your files locally?
>>
>
> Here is every downside I can come up with.  Some are trivial, but I've
> tried to play devil's advocate.
>
> 1) Additional memory/processing footprint, as you mentioned.
>
> 2) Additional network usage by Kafka consumers hitting producer box.
>
> 3) Loss of either producer elasticity or message retention.  One of
> the awesome features of Kafka is being able to hold on a long history
> of messages and replay as needed.  But if the producers machines hold
> my brokers, I can no longer scale down the number of producer machines
> as the system load drops--a loss of elasticity.  This ties to another
> group discussion about decommissioning brokers.
>
> 4) Security... some producers live on web-accessible machines.  The
> less open ports accepting incoming connections, the better.  Not a
> huge issue with good firewall rules, but still something to ponder.
>
> 5) Loss of data duplication has downsides too.  Having multiple copies
> of data does violate DRY, but it can also add robustness.  If the
> hard-disk on either the producer or the brokers dies, you still have
> the data lying around.  Since the data is not getting changed on
> either the broker or any producer log files, you shouldn't have
> syncing issues normally associated with DRY violations.
>
>
> - Niek

Mime
View raw message