kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Only one broker has partitions
Date Fri, 17 Feb 2012 22:23:02 GMT
I see the problem. This is a bug in Kafka. Basically, for a new topic, we
bootstrap using all existing brokers. However, if a topic already exists on
some brokers, we never bootstrap again, which means new brokers will be
ignored. Could you file a jira for that? For now, you have to manually
create the topic on the new brokers (e.g., by sending some data to the new
broker directly).

Thanks,

Jun


On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <mabateman@ebay.com> wrote:

> Hi Jun,
>
> Thanks for the details. What sort of time frame is eventually?
>
> In the case of a topic that exists before nodes are added, will the other
> nodes ever get that topic? We've been running all three for about a week
> now it's still only on our 0 broker.
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 17, 2012 11:19 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> Matt,
>
> The behavior is a bit weird for a new topic. Initially, non of the brokers
> has the topic. So, the producer simply assumes there is 1 partition per
> broker. Once a topic is created in a broker, the broker registers
> #partitions for that topic in ZK. Then the producer will pick up the new
> #partitions for that broker and refresh its list of available partitions.
> Eventually, the topic will be created on every broker and the producer
> should see a total of 60 partitions.
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <mabateman@ebay.com>
> wrote:
>
> > We're using a single topic ("test-topic") in our QA setup. Originally
> > we were using only one broker (id 0) in or testing environment.
> > Recently we added two more (id 2 and 3) for a total of 3 in our
> > cluster. ZooKeeper shows that the topic and partitions are registered by
> broker 0 as expected.
> > They do not show for brokers 2 and 3. I'm guessing that if a broker is
> > not registered with ZooKeeper when a producer sends the first message
> > for a topic, it will never process messages for that topic?
> >
> > As a test, I sent messages into the cluster for a new topic. The
> > producer logs show
> >
> > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered
> > under
> > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test with
> > available brokers in the cluster with default number of partitions = 1
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Adding following broker id, partition
> > id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
> >
> > This seems logical, all brokers are mentioned. Then I see the
> > following (each broker is configured to use 20 partitions):
> >
> > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> > (kafka.producer.ProducerPool)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of topics
> > changed at /brokers/topics Updated topics -> [topic1, topic2,
> > cluster-test, topic3, test-topic, topic4]
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of
> topics:
> > Set(topic1, topic3, topic2, test-topic, topic4)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list of
> > topics: Set(topic4, test-topic, topic2, topic3, cluster-test, topic1)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly
> > registered topics: Set(cluster-test)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> > registered list of brokers for topic: cluster-test are Buffer(2)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered
> > list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of broker
> > partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1, 2-2,
> > 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15,
> > 2-16, 2-17, 2-18, 2-19, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8,
> > 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> > (kafka.producer.Producer)
> >
> > This seems strange, shouldn't each broker host each partition for the
> > new "cluster-topic"?
> >
> > I then see the producer debug messages showing messages sent to broker 2.
> >
> > What am I missing here?
> >
> > Thanks,
> >
> > Matt
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, February 16, 2012 6:08 PM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Only one broker has partitions
> >
> > Try enabling debug level logging in Producer class. It will show which
> > broker the partitioner selects.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <mabateman@ebay.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I feel like this was discussed at some point but couldn't find it in
> > > the docs or the mailing list archives, hopefully this isn't redundant.
> > >
> > > We are running 3 version 0.7 brokers using an ensemble of 3
> > > ZooKeeper nodes. We use a ZooKeeper connect string in the producers
> and consumers.
> > > ZooKeeper is enabled on each broker. We use the default partitioner
> > > in the producer supplying it an integer. Each broker has a unique Id.
> > >
> > > However, only one broker has any log files. The log from that
> > > machine shows it registering the topics in ZooKeeper. The other
> > > brokers don't have any log files.
> > >
> > > Have a I missed a basic configuration setting or misunderstood
> > > something operational? The only slight mismatch I see is we run
> > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message