kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <stevenz...@gmail.com>
Subject Re: [DISCUSSION] Boot dependency in the new producer
Date Tue, 27 Jan 2015 00:42:13 GMT
Jay, I don't think this line will bootstrap full metadata (for all topics).
it will just construct the cluster object with bootstrap host. you need to
do "metadata.add(topic)"  to set interest of a topic's partition metadata.

Guozhang, I personally think this is ok. it just do a few DNS lookup or TCP
connection before first send.

On Mon, Jan 26, 2015 at 2:07 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> Oh, yes, I guess I thought you meant that construction of the client would
> block on the metadata request.
>
> I don't personally think that is a problem because if it fails it will
> retry in the background, right?
>
> But actually I think this is probably violating another desirable criteria
> we had talked about which was keeping the producer from bootstrapping the
> full metadata for all partitions. If it is doing that during construction
> time presumably the resulting metadata request is for all partitions, no?
> That isn't a huge problem, but I think isn't what was intended.
>
> -Jay
>
> On Mon, Jan 26, 2015 at 1:34 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
> > It will set the needUpdate flag to true and hence the background Sender
> > will try to talk to the bootstrap servers.
> >
> > Guozhang
> >
> > On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
> >
> > > Hey Guozhang,
> > >
> > > That line shouldn't cause any connections to Kafka to be established,
> > does
> > > it? All that is doing is creating the Cluster pojo using the supplied
> > > addresses. The use of InetSocketAddress may cause some dns stuff to
> > happen,
> > > though...
> > >
> > > -Jay
> > >
> > > On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang <wangguoz@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am not sure if we have discussed about this before, but recently I
> > > > realized that we have introduced boot dependency of the kafka-server
> > > > specified by the "bootstrap.servers" config in the new producer. More
> > > > specifically, although in the old producer we also have a similar
> > config
> > > > for specifying the broker list, the producer will not try to connect
> to
> > > > those brokers until the first message send call is triggered; whereas
> > in
> > > > the new producer, it will try to talk to them in construction time
> via:
> > > >
> > > > update(Cluster.bootstrap(addresses), time.milliseconds());
> > > >
> > > >
> > > > I personally am neutral to this change, as in most cases the
> > > corresponding
> > > > kafka server should be up and running before the producer clients are
> > > > deployed, but there are still some corner cases when it is not true,
> > for
> > > > example some standalone deployment tests of the app embedded with
> some
> > > > clients, etc. So I would like to bring this up to people's attention
> if
> > > we
> > > > have not discussed about it before: do we think this is OK to
> introduce
> > > > this boot dependency in the new producer?
> > > >
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message