kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <n...@confluent.io>
Subject Re: [VOTE] 0.8.2.0 Candidate 3
Date Sun, 01 Feb 2015 21:12:49 GMT
+1 (binding). Verified quickstart and unit tests ran ok.

On Sun, Feb 1, 2015 at 9:11 AM, Steven Wu <stevenz3wu@gmail.com> wrote:

> In Netflix, we have been using route53 DNS name as bootstrap servers in AWS
> env. Basically, when a kafka broker start, we add it to route53 DNS name
> for the cluster. this is like the VIP that Jay suggested.
>
> But we are also moving toward to use Eureka service registry for
> bootstrapping. We are worried that if DNS name happens to resolve to a bad
> broker. it might impact the bootstrap process/resiliency. We want to get a
> list of brokers from Eureka to pass in as "bootstrap.servers".
>
>
>
> On Sun, Feb 1, 2015 at 5:30 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>
> > You may already know this but the producer doesn't require a complete
> list
> > of brokers in its config, it just requires the connection info for one
> > active broker which it uses to discover the rest of the brokers. We allow
> > you to specify multiple urls here for failover in cases where you aren't
> > using a vip. So if you can put three brokers into the VIP for metadata
> > bootstrapping you can still scale up and down the rest of the cluster.
> >
> > -Jay
> >
> > On Sun, Feb 1, 2015 at 12:17 AM, Alex The Rocker <alex.m3tal@gmail.com>
> > wrote:
> >
> > > Jun:
> > >
> > > You raise a very good question: let me explain why we use
> > > Broker.getConnectionString(), so may be we'll get a supported way to
> > > answer our need.
> > >
> > > We use Broker.getConnectionString() because we deploy "Kafka services"
> > > in Amazon EC2 with the following architecture:
> > > * Three VMs dedicated to Zookeeper processes
> > > * At least two VMs with Kafka broker, but depending on load it can be
> > > scaled to more broker VMs. Brokers self-register their address in
> > > Zookeeper by serializing Broker objects in Zk.
> > >
> > > The VMs with Zookeeper have Elastic IPs = stable public IPs,
> > >
> > > These public IPs are fed to the  various "Application services" which
> > > rely on Kafka to stream their logs & monitoring data to our central
> > > Hadoop system.
> > >
> > > Using zkclient and the above mentionned public zookeeper IPs, we get
> > > the list of brokers registrered to a given "Kafka service":  this is
> > > where we unserializer Broker objects and then use
> > > getConnectionString() to discover the brokers' addresses. Then,
> > > brokers addresses are used to initialize the Kafka producer(s).
> > >
> > > The whole trick is that we cannot use Elastic IP (=stable IPs) for
> > > Kafka VMs, because of their 'elastic" nature : we want to be able to
> > > scale up / down the number of VMs with Kafka brokers.
> > >
> > > Now, we understand that using non public Kafka API is bad : we've been
> > > broken when moving to 0.8.1.1, then again when moving to 0.8.2.0...
> > >
> > > So it's time to raise the right question: what would be the supported
> > > way to configure our producers given our dynamic-IP-for-brokers
> > > context?
> > >
> > > Thanks,
> > > Alex.
> > >
> > > 2015-02-01 8:55 GMT+01:00 VERMEERBERGEN Alexandre
> > > <Alexandre.VERMEERBERGEN@3ds.com>:
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:jun@confluent.io]
> > > > Sent: Sunday, February 01, 2015 3:03
> > > > To: users@kafka.apache.org; kafka-clients@googlegroups.com
> > > > Cc: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] 0.8.2.0 Candidate 3
> > > >
> > > > Hi, Alex,
> > > >
> > > > Thanks for testing RC3.
> > > >
> > > > Broker.connectionString() is actually not part of the public api for
> > the
> > > producer. Is there a particular reason that you need to use this api?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Sat, Jan 31, 2015 at 1:53 PM, Alex The Rocker <
> alex.m3tal@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> I have read Broker.scala source code, and I found the answer:
> > > >>  - With Kafka 0.8.1.1 we used Broker.getConnectionString() in our
> Java
> > > >> code.
> > > >>  - With Kafka 0.8.2.0, this method has been replaced by a 0-arity
> > > >> method without the "get" prefix, so we have to change our Java code
> to
> > > >> call
> > > >> Broker.connectionString()
> > > >>
> > > >> So despite binary compatibility is broken, we have a by-pass.
> > > >> I hope this will help other people relying on this API...
> > > >>
> > > >> and I'm going to continue tests with 0.8.2 rc3..
> > > >>
> > > >> Alex
> > > >>
> > > >> 2015-01-31 21:23 GMT+01:00 Alex The Rocker <alex.m3tal@gmail.com>:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > I ran my own tests made with kafka_2.10-0.8.1.1.tgz binaries
with
> > > >> > our
> > > >> > application:
> > > >> >
> > > >> > 1st test:
> > > >> > ======
> > > >> >   replace all kafka .jar files in our application on consumming
> side
> > > >> >   (without recompiling anything)
> > > >> >   => tests passed, OK
> > > >> >
> > > >> > 2nd test:
> > > >> > =======
> > > >> >   replace all kafka .jar files in our application on producubg
> side
> > > >> >   (without recompiling anything)
> > > >> >   => KO, we get this error:
> > > >> >
> > > >> > 2015-01-31 20:54:00,094 [Timer-2] ERROR c.d.i.t.StdOutErrRedirect
> -
> > > >> > Exception in thread "Timer-2"
> > > >> > 2015-01-31 20:54:00,111 [Timer-2] ERROR c.d.i.t.StdOutErrRedirect
> -
> > > >> > java.lang.NoSuchMethodError:
> > > >> > kafka.cluster.Broker.getConnectionString()Ljava/lang/String;
> > > >> >
> > > >> > Which means that binary compatibility with 0.8.1.1 version has
> been
> > > >> broken.
> > > >> > We use getConnectionString() to get Broker's zookeepers adresses,
> > > >> > see
> > > >> this
> > > >> > answer from Neha:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > http://mail-archives.apache.org/mod_mbox/kafka-users/201404.mbox/%3CCA
> > > >> OG_4QYnWrB=TmRTCRyF8-PdAgY_CGFe_cxoTQBCLrKj2+XmwA@mail.gmail.com%3E
> > > >> >
> > > >> > If the kafka.cluster.Broker.getConnectionString() method has
been
> > > >> > removed with Kafka 0.8.2.0, then what is the suitable replacement
> > for
> > > it ?
> > > >> >
> > > >> > Thanks
> > > >> > Alex
> > > >> >
> > > >> >
> > > >> >> -----Original Message-----
> > > >> >> From: Jun Rao [mailto:jun@confluent.io]
> > > >> >> Sent: Thursday, January 29, 2015 6:22
> > > >> >> To: dev@kafka.apache.org; users@kafka.apache.org;
> > > >> >> kafka-clients@googlegroups.com
> > > >> >> Subject: [VOTE] 0.8.2.0 Candidate 3
> > > >> >>
> > > >> >> This is the third candidate for release of Apache Kafka 0.8.2.0.
> > > >> >>
> > > >> >> Release Notes for the 0.8.2.0 release
> > > >> >>
> > > >> >>
> > > >>
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/RELEASE_NOT
> > > >> ES.html
> > > >> >>
> > > >> >> *** Please download, test and vote by Saturday, Jan 31, 11:30pm
> PT
> > > >> >>
> > > >> >> Kafka's KEYS file containing PGP keys we use to sign the
release:
> > > >> >> http://kafka.apache.org/KEYS in addition to the md5, sha1
and
> sha2
> > > >> >> (SHA256) checksum.
> > > >> >>
> > > >> >> * Release artifacts to be voted upon (source and binary):
> > > >> >> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/
> > > >> >>
> > > >> >> * Maven artifacts to be voted upon prior to release:
> > > >> >> https://repository.apache.org/content/groups/staging/
> > > >> >>
> > > >> >> * scala-doc
> > > >> >>
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/scaladoc
> > > >> >> /
> > > >> >>
> > > >> >> * java-doc
> > > >> >>
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/javadoc/
> > > >> >>
> > > >> >> * The tag to be voted upon (off the 0.8.2 branch) is the
0.8.2.0
> > > >> >> tag
> > > >> >>
> > > >>
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=223ac42a7a
> > > >> 2a0dab378cc411f4938a9cea1eb7ea
> > > >> >> (commit 7130da90a9ee9e6fb4beb2a2a6ab05c06c9bfac4)
> > > >> >>
> > > >> >> /*******************************************
> > > >> >>
> > > >> >> Thanks,
> > > >> >>
> > > >> >> Jun
> > > >> >>
> > >
> >
>



-- 
Thanks,
Neha

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message