kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: consumer not consuming messages
Date Fri, 11 Apr 2014 04:34:33 GMT
Hi Ajrun,

It seems to be the cause:

https://issues.apache.org/jira/browse/KAFKA-1016

Guozhang



On Thu, Apr 10, 2014 at 9:21 PM, Arjun <arjun@socialtwist.com> wrote:

> I hope this one would give u  a better idea.
>
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
> --zkconnect zkhost:port --topic testtopic
> Group           Topic                          Pid Offset logSize
> Lag             Owner
> group1          testtopic    0   253             253 0
> group1_ip-xx-1397188061429-b5ff1205-0
> group1          testtopic    1   267             267 0
> group1_ip-xx-1397188061429-b5ff1205-0
> group1          testtopic    2   254             254 0
> group1_ip-xx-1397188061429-b5ff1205-0
> group1          testtopic    3   265             265 0
> group1_ip-xx-1397188061429-b5ff1205-0
> group1          testtopic    4   261             261 0
> group1_ip-xx-1397188061429-b5ff1205-1
> group1          testtopic    5   294             294 0
> group1_ip-xx-1397188061429-b5ff1205-1
> group1          testtopic    6   248             248 0
> group1_ip-xx-1397188061429-b5ff1205-1
> group1          testtopic    7   271             271 0
> group1_ip-xx-1397188061429-b5ff1205-1
> group1          testtopic    8   240             240 0
> group1_ip-xx-1397188061429-b5ff1205-2
> group1          testtopic    9   261             261 0
> group1_ip-xx-1397188061429-b5ff1205-2
> group1          testtopic    10  290             290 0
> group1_ip-xx-1397188061429-b5ff1205-2
> group1          testtopic    11  250             251 1
> group1_ip-xx-1397188061429-b5ff1205-2
>
> If you see the output, in the last line the lag is 1 for that partition. I
> just send one message. This topic is not new as you see there are lot of
> messages which have accumlated from yesterday. This one message will not be
> consumed by consumer what so ever. But if i send some 10 messages then all
> the messages are consumed.
>
> Please let me know if i have to change any consumer properties.
>
> My consumer properties are :
> "fetch.wait.max.ms"="180000"
> "fetch.min.bytes" = "1"
> "auto.offset.reset" = "smallest"
> "auto.commit.enable"=  "false"
> "fetch.message.max.bytes" = "1048576"
>
>
> Thanks
> Arjun Narasimha Kota
> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
>
>>
>> The consumer uses do specific topics.
>>
>> On Apr 11, 2014 6:23 AM, "Arjun Kota" <arjun@socialtwist.com <mailto:
>> arjun@socialtwist.com>> wrote:
>>
>>     Yes the message shows up on the server.
>>
>>     On Apr 11, 2014 12:07 AM, "Guozhang Wang" <wangguoz@gmail.com
>>     <mailto:wangguoz@gmail.com>> wrote:
>>
>>         Hi Arjun,
>>
>>         If you only send one message, does that message show up on the
>>         server? Does
>>         you consumer use wildcard topics or specific topics?
>>
>>         Guozhang
>>
>>
>>         On Thu, Apr 10, 2014 at 9:20 AM, Arjun <arjun@socialtwist.com
>>         <mailto:arjun@socialtwist.com>> wrote:
>>
>>         > But  we have auto offset reset to smallest not largest, even
>>         then this
>>         > issue arises? If so is there any work around?
>>         >
>>         > Thanks
>>         > Arjun NArasimha Kota
>>         >
>>         >
>>         > On Thursday 10 April 2014 09:39 PM, Guozhang Wang wrote:
>>         >
>>         >> It could be https://issues.apache.org/jira/browse/KAFKA-1006.
>>         >>
>>         >> Guozhang
>>         >>
>>         >>
>>         >> On Thu, Apr 10, 2014 at 8:50 AM, Arjun
>>         <arjun@socialtwist.com <mailto:arjun@socialtwist.com>> wrote:
>>         >>
>>         >>  its auto created
>>         >>> but even after topic creation this is the scenario
>>         >>>
>>         >>> Arjun
>>         >>>
>>         >>> On Thursday 10 April 2014 08:41 PM, Guozhang Wang wrote:
>>         >>>
>>         >>>  Hi Arjun,
>>         >>>>
>>         >>>> Did you manually create the topic or use auto.topic.creation?
>>         >>>>
>>         >>>> Guozhang
>>         >>>>
>>         >>>>
>>         >>>> On Thu, Apr 10, 2014 at 7:39 AM, Arjun
>>         <arjun@socialtwist.com <mailto:arjun@socialtwist.com>> wrote:
>>         >>>>
>>         >>>>   Hi,
>>         >>>>
>>         >>>>> We have 3 node kafka 0.8 setup with zookeepers ensemble.
>>         We use high
>>         >>>>> level
>>         >>>>> consumer with auto commit offset false. I am facing
some
>>         peculiar
>>         >>>>> problem
>>         >>>>> with kafka. When i send some 10-20 messages or so the
>>         consumer starts
>>         >>>>> to
>>         >>>>> consume the messages. But if  i send only one message
to
>>         kafka, then
>>         >>>>> even
>>         >>>>> though consumer is active it is not trying to fetch
the
>>         message. There
>>         >>>>> is
>>         >>>>> nothing in logs, just the messages are being fetched
by
>>         the kafka
>>         >>>>> consumer.
>>         >>>>> The messages are there in the Kafka server. Can some
one
>>         let me know
>>         >>>>> where
>>         >>>>> i am doing wrong.
>>         >>>>>
>>         >>>>>
>>         >>>>> Thanks
>>         >>>>> Arjun Narasimha Kota
>>         >>>>>
>>         >>>>>
>>         >>>>>
>>         >>>>
>>         >>
>>         >
>>
>>
>>         --
>>         -- Guozhang
>>
>>
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message