kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dick Davies <d...@hellooperator.net>
Subject Re: Kafka + ZooKeeper on the same hardware?
Date Mon, 18 Jan 2016 20:21:32 GMT
Started an Ansible playbook using the Confluent platform RPM distro,
and it seems that co-locates zookeepers
on the brokers.

So I'm assuming it's fine (at least on 0.9.x for the reasons Todd mentioned).

Does anyone know if the Confluent distro is supposed to be production-ready?
I switched our test setup over to the Confluent distro to get
RPM-based installs etc and got a 3-node
cluster up in HA, but there were a few things I had to roll myself
(init scripts etc) which I'd have expected
in a full release.

(sorry if this is OT, I don't know of a specific Confluent mailing list).



On 14 January 2016 at 18:31, Todd Palino <tpalino@gmail.com> wrote:
> I’d say it depends on load and usage. It can definitely be done, and we’ve
> done it here in places, though we don’t anymore. Part of the luxury of
> being able to get the hardware we want. In general, it’s probably easier to
> do with 0.9 and Kafka-committed offsets, because the consumers don’t need
> to talk to ZK as much. It’s probably even even easier with the new
> consumer, but I can’t speak to that at all.
>
> One of the gotchas is that ZK really should have its transaction log on an
> isolate device so that sequential writes do not require seeks. This could
> be a separate disk or an SSD drive. An example of a really bad place to put
> it would be on the same device as your Kafka log segments :) Depending on
> your load, it may not be critical to use a separate device.
>
> As Gwen noted, it all comes down to load. Your availability will be fine,
> you just need to figure out if the services can share the load.
>
> -Todd
>
>
> On Thu, Jan 14, 2016 at 9:25 AM, Gwen Shapira <gwen@confluent.io> wrote:
>
>> It depends on load :)
>> As long as there is no contention, you are fine.
>>
>> On Thu, Jan 14, 2016 at 6:06 AM, Erik Forsberg <forsberg@opera.com> wrote:
>>
>> > Hi!
>> >
>> > Pondering how to configure Kafka clusters and avoid having too many
>> > machines to manage.. Would it be recommended to run say a 3 node kafka
>> > cluster where you also run your 3 node zookeeper cluster on the same
>> > machines?
>> >
>> > I guess the answer is that "it depends on load", but would be interested
>> > in any opinions on this anyway.
>> >
>> > Thanks!
>> > \EF
>> >
>>
>
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino

Mime
View raw message