kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Only one broker has partitions
Date Fri, 17 Feb 2012 23:03:02 GMT
Thanks, Bateman.

Jun

On Fri, Feb 17, 2012 at 2:58 PM, Bateman, Matt <mabateman@ebay.com> wrote:

> Hi Jun,
>
> https://issues.apache.org/jira/browse/KAFKA-278
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 17, 2012 2:23 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> I see the problem. This is a bug in Kafka. Basically, for a new topic, we
> bootstrap using all existing brokers. However, if a topic already exists on
> some brokers, we never bootstrap again, which means new brokers will be
> ignored. Could you file a jira for that? For now, you have to manually
> create the topic on the new brokers (e.g., by sending some data to the new
> broker directly).
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <mabateman@ebay.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the details. What sort of time frame is eventually?
> >
> > In the case of a topic that exists before nodes are added, will the
> > other nodes ever get that topic? We've been running all three for
> > about a week now it's still only on our 0 broker.
> >
> > Thanks,
> >
> > Matt
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Friday, February 17, 2012 11:19 AM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Only one broker has partitions
> >
> > Matt,
> >
> > The behavior is a bit weird for a new topic. Initially, non of the
> > brokers has the topic. So, the producer simply assumes there is 1
> > partition per broker. Once a topic is created in a broker, the broker
> > registers #partitions for that topic in ZK. Then the producer will
> > pick up the new #partitions for that broker and refresh its list of
> available partitions.
> > Eventually, the topic will be created on every broker and the producer
> > should see a total of 60 partitions.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <mabateman@ebay.com>
> > wrote:
> >
> > > We're using a single topic ("test-topic") in our QA setup.
> > > Originally we were using only one broker (id 0) in or testing
> environment.
> > > Recently we added two more (id 2 and 3) for a total of 3 in our
> > > cluster. ZooKeeper shows that the topic and partitions are
> > > registered by
> > broker 0 as expected.
> > > They do not show for brokers 2 and 3. I'm guessing that if a broker
> > > is not registered with ZooKeeper when a producer sends the first
> > > message for a topic, it will never process messages for that topic?
> > >
> > > As a test, I sent messages into the cluster for a new topic. The
> > > producer logs show
> > >
> > > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker
> > > partitions registered for topic: cluster-test
> > > (kafka.producer.Producer)
> > > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered
> > > under
> > > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test
> > > with available brokers in the cluster with default number of
> > > partitions = 1
> > > (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,705] DEBUG Adding following broker id,
> > > partition id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> > > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
> > >
> > > This seems logical, all brokers are mentioned. Then I see the
> > > following (each broker is configured to use 20 partitions):
> > >
> > > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> > > (kafka.producer.ProducerPool)
> > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of
> > > topics changed at /brokers/topics Updated topics -> [topic1, topic2,
> > > cluster-test, topic3, test-topic, topic4]
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of
> > topics:
> > > Set(topic1, topic3, topic2, test-topic, topic4)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list
> > > of
> > > topics: Set(topic4, test-topic, topic2, topic3, cluster-test,
> > > topic1)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly
> > > registered topics: Set(cluster-test)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> > > registered list of brokers for topic: cluster-test are Buffer(2)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered
> > > list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of
> > > broker partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1,
> > > 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13,
> > > 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker
> > > partitions registered for topic: cluster-test
> > > (kafka.producer.Producer)
> > > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> > > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7,
> > > 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18,
> > > 2-19, 3-0)
> > > (kafka.producer.Producer)
> > >
> > > This seems strange, shouldn't each broker host each partition for
> > > the new "cluster-topic"?
> > >
> > > I then see the producer debug messages showing messages sent to broker
> 2.
> > >
> > > What am I missing here?
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, February 16, 2012 6:08 PM
> > > To: kafka-users@incubator.apache.org
> > > Subject: Re: Only one broker has partitions
> > >
> > > Try enabling debug level logging in Producer class. It will show
> > > which broker the partitioner selects.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <mabateman@ebay.com>
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I feel like this was discussed at some point but couldn't find it
> > > > in the docs or the mailing list archives, hopefully this isn't
> redundant.
> > > >
> > > > We are running 3 version 0.7 brokers using an ensemble of 3
> > > > ZooKeeper nodes. We use a ZooKeeper connect string in the
> > > > producers
> > and consumers.
> > > > ZooKeeper is enabled on each broker. We use the default
> > > > partitioner in the producer supplying it an integer. Each broker has
> a unique Id.
> > > >
> > > > However, only one broker has any log files. The log from that
> > > > machine shows it registering the topics in ZooKeeper. The other
> > > > brokers don't have any log files.
> > > >
> > > > Have a I missed a basic configuration setting or misunderstood
> > > > something operational? The only slight mismatch I see is we run
> > > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> > > >
> > > > Thanks,
> > > >
> > > > Matt
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message