kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: One of two consumers is always Idle though I have 2 partitions
Date Fri, 14 Nov 2014 03:34:59 GMT
Yup, sounds like
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?

This should go away with 0.8.2 with the default partions now being 1 =8^)
with auto create topics.

/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
On Nov 13, 2014 8:34 PM, "Chia-Chun Shih" <chiachun.shih@gmail.com> wrote:

> Hi Palur,
>
> When producing messages, did you specify a key in your KeyedMessage? If
> not, producer will send all messages to ONE randomly selected partition and
> stick to this partition for 10 minutes by default.
>
> regards,
> Chia-Chun
>
> 2014-11-14 7:19 GMT+08:00 Jagat Singh <jagatsingh@gmail.com>:
>
> > It would be worth reading once the consumer section from the
> documentation.
> >
> > https://kafka.apache.org/documentation.html
> >
> >
> >
> > On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep <psandeep@hawk.iit.edu>
> > wrote:
> >
> > > Yes, they are on the same consumer group, but I have two partitions.
> > >
> > > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh <jagatsingh@gmail.com>
> > wrote:
> > >
> > > > Are both of them in same Consumer Group?
> > > >
> > > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep <
> psandeep@hawk.iit.edu>
> > > > wrote:
> > > >
> > > > > Dear Developers,
> > > > >
> > > > > I am 2nd year masters student at IIT. I am using Kafka for one of
> my
> > > > > research projects.My question is the following:
> > > > >
> > > > > 1. I have a producer, consumer and a broker(that contains 1st
> > partition
> > > > of
> > > > > my topic)  on node1
> > > > > 2. I have a producer, consumer, zookeeper and a broker(that
> contains
> > > 2nd
> > > > > partition of my topic)  on node2
> > > > > 3. Here comes my problem: though I have two partitions only one
> > > consumer
> > > > > pulls messages and the other one is always idle.
> > > > >
> > > > > What is that I can do to keep both of my consumer busy?
> > > > >
> > > > > Thank you
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Sandeep Palur
> > > > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > > > Department of Computer Science, Illinois Institute of Technology
> > (IIT)
> > > > > Phone : 312-647-9833
> > > > > Email : psandeep@hawk.iit.edu <srajasur@hawk.iit.edu>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Sandeep Palur
> > > Data-Intensive Distributed Systems Laboratory, CS/IIT
> > > Department of Computer Science, Illinois Institute of Technology (IIT)
> > > Phone : 312-647-9833
> > > Email : psandeep@hawk.iit.edu <srajasur@hawk.iit.edu>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message