kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Broker ID
Date Fri, 09 Dec 2011 17:46:17 GMT
I think what Florian is saying is actually subtly different. Usually people
ask, "Do you really need a node id? why not just use host/port so I don't
need to allocate an id?" As Jun says you need a node id so that data is
portable and not tied to the host/port of the machine serving it. But if we
rephrase what Florian is saying as a feature request I would put it this
way, it is kind of a pain to have to allocate ids "by hand" and this makes
it harder to add/remove brokers easily. The right id to assign, after all,
is always the next unused integer. It could instead be that node ids are
auto-allocated when the broker first starts and stored with the data. The
mechanism would be a file in the data directory called .node-id. If this
file is absent the broker would allocate itself the next available id in zk
in a transactional way and store it in the .node-id file and assume that
node id from then on. This would make dynamic deployment on ec2 or for
integration/perf tests much easier, and would likely ease our own
configuration as well since we would not have any per-machine config
overrides.

The question is, how would this interact with replication...?

-Jay

On Fri, Dec 9, 2011 at 9:37 AM, Jun Rao <junrao@gmail.com> wrote:

> Florian,
>
> The broker id uniquely identifies a broker. Every time a broker is
> restarted, we want the broker to come up with the same id. The consumer
> relies on the broker id to resume consumption properly.
>
> Thanks,
>
> Jun
>
> On Fri, Dec 9, 2011 at 8:34 AM, Florian Leibert <flo@leibert.de> wrote:
>
> > Hi -
> > I see there's a static broker-id in the properties files. Is this
> optional?
> > When using Zookeeper, couldn't ZK just give each broker if registered as
> a
> > SEQ node a generated ID that's guaranteed to be unique? Otherwise can
> this
> > ID be a randomly generated ID? Basically I'd like to bring up instances
> in
> > EC2 without having to worry about the state of other brokers...
> >
> > Thanks!
> > Flo
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message