kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Consumer thread dies
Date Tue, 11 Nov 2014 05:42:00 GMT
Don't you have the same problem using SimpleConsumer? How does another
process know a SimpleConsumer hangs?

Thanks,

Jun

On Mon, Nov 10, 2014 at 9:47 AM, Srinivas Reddy Kancharla <
getreddy@gmail.com> wrote:

> Thanks Jun for your response.
> Here is my scenario:
>
> topicCountMap.put(topic, new Integer(2));
> Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =
> consumer.createMessageStreams(topicCountMap);
> List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
>
> So from above scenario (only 1 partition) , there will be 2 threads C1 and
> C2, and one of the thread would be holding one partition. In my scenario,
> C1 would be hung after consuming the message from stream and in hung state
> during processing of message. So in such scenario, looks like C2 will not
> get hold of partition stream unless C1 didn't die..
>
> So considering such scenarios, I am planning to use SimpleConsumer where I
> will have more control on consuming  Partition content. But I have to
> maintain offset state in zookeeper. Let me know if this approach is correct
> for such hung scenarios.
>
> Thanks and regards,
> Srini
>
>
> On Sun, Nov 9, 2014 at 9:12 PM, Jun Rao <junrao@gmail.com> wrote:
>
> > If C1 dies, C2 will be owning that partition. However, C1 has to really
> > die, which typically means that either you close the consumer connector
> or
> > the jvm of C1 is gone.
> >
> > In your case, it seems that C1 didn't die, it just hung. Do you know why
> C1
> > hung?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Nov 7, 2014 at 3:34 PM, Srinivas Reddy Kancharla <
> > getreddy@gmail.com
> > > wrote:
> >
> > > Hi,
> > >
> > > I have a scenario where I have 1 partition and 1 consumer group having
> 2
> > > consumer threads running say C1 and C2. Since there is only one
> partition
> > > for a given topic, say C1 is holding that partition. Now due to some
> > reason
> > > if C1 dies, can C2 get hold of that partition?
> > >
> > > i.e. C1 was busy with KafkaStream instance, for any reason if C1 dies
> or
> > in
> > > hung state, Can we make C2 talking to KafkaStream (for Partition 0).
> > > I am facing this issue where I have 10 messages in partition 0 and C1
> was
> > > consuming it. At message 4, C1 went into hung state. Now I would like
> to
> > > make C2 to consumer other messages which are not consumed by C1.
> > >
> > > Thank and regards,
> > > Srini
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message