kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bateman, Matt" <mabate...@ebay.com>
Subject RE: Only one broker has partitions
Date Fri, 17 Feb 2012 22:58:20 GMT
Hi Jun,

https://issues.apache.org/jira/browse/KAFKA-278

Thanks,

Matt

-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Friday, February 17, 2012 2:23 PM
To: kafka-users@incubator.apache.org
Subject: Re: Only one broker has partitions

I see the problem. This is a bug in Kafka. Basically, for a new topic, we bootstrap using
all existing brokers. However, if a topic already exists on some brokers, we never bootstrap
again, which means new brokers will be ignored. Could you file a jira for that? For now, you
have to manually create the topic on the new brokers (e.g., by sending some data to the new
broker directly).

Thanks,

Jun


On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <mabateman@ebay.com> wrote:

> Hi Jun,
>
> Thanks for the details. What sort of time frame is eventually?
>
> In the case of a topic that exists before nodes are added, will the 
> other nodes ever get that topic? We've been running all three for 
> about a week now it's still only on our 0 broker.
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 17, 2012 11:19 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> Matt,
>
> The behavior is a bit weird for a new topic. Initially, non of the 
> brokers has the topic. So, the producer simply assumes there is 1 
> partition per broker. Once a topic is created in a broker, the broker 
> registers #partitions for that topic in ZK. Then the producer will 
> pick up the new #partitions for that broker and refresh its list of available partitions.
> Eventually, the topic will be created on every broker and the producer 
> should see a total of 60 partitions.
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <mabateman@ebay.com>
> wrote:
>
> > We're using a single topic ("test-topic") in our QA setup. 
> > Originally we were using only one broker (id 0) in or testing environment.
> > Recently we added two more (id 2 and 3) for a total of 3 in our 
> > cluster. ZooKeeper shows that the topic and partitions are 
> > registered by
> broker 0 as expected.
> > They do not show for brokers 2 and 3. I'm guessing that if a broker 
> > is not registered with ZooKeeper when a producer sends the first 
> > message for a topic, it will never process messages for that topic?
> >
> > As a test, I sent messages into the cluster for a new topic. The 
> > producer logs show
> >
> > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker 
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered 
> > under
> > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test 
> > with available brokers in the cluster with default number of 
> > partitions = 1
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Adding following broker id, 
> > partition id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
> >
> > This seems logical, all brokers are mentioned. Then I see the 
> > following (each broker is configured to use 20 partitions):
> >
> > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> > (kafka.producer.ProducerPool)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of 
> > topics changed at /brokers/topics Updated topics -> [topic1, topic2, 
> > cluster-test, topic3, test-topic, topic4]
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of
> topics:
> > Set(topic1, topic3, topic2, test-topic, topic4)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list 
> > of
> > topics: Set(topic4, test-topic, topic2, topic3, cluster-test, 
> > topic1)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly 
> > registered topics: Set(cluster-test)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently 
> > registered list of brokers for topic: cluster-test are Buffer(2)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered 
> > list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of 
> > broker partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1, 
> > 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 
> > 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker 
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 
> > 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 
> > 2-19, 3-0)
> > (kafka.producer.Producer)
> >
> > This seems strange, shouldn't each broker host each partition for 
> > the new "cluster-topic"?
> >
> > I then see the producer debug messages showing messages sent to broker 2.
> >
> > What am I missing here?
> >
> > Thanks,
> >
> > Matt
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, February 16, 2012 6:08 PM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Only one broker has partitions
> >
> > Try enabling debug level logging in Producer class. It will show 
> > which broker the partitioner selects.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <mabateman@ebay.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I feel like this was discussed at some point but couldn't find it 
> > > in the docs or the mailing list archives, hopefully this isn't redundant.
> > >
> > > We are running 3 version 0.7 brokers using an ensemble of 3 
> > > ZooKeeper nodes. We use a ZooKeeper connect string in the 
> > > producers
> and consumers.
> > > ZooKeeper is enabled on each broker. We use the default 
> > > partitioner in the producer supplying it an integer. Each broker has a unique
Id.
> > >
> > > However, only one broker has any log files. The log from that 
> > > machine shows it registering the topics in ZooKeeper. The other 
> > > brokers don't have any log files.
> > >
> > > Have a I missed a basic configuration setting or misunderstood 
> > > something operational? The only slight mismatch I see is we run 
> > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> >
>

Mime
View raw message