kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: consumer not consuming messages
Date Fri, 11 Apr 2014 15:11:31 GMT
We may have a bug that doesn't observe etch.min.bytes accurately. So a
lower fetch.wait.max.ms will improve consumer latency.

Could you run a console consumer and see if you have the same issue? That
will tell us if this is a server side issue or an issue just in your
consumer.

Thanks,

Jun


On Thu, Apr 10, 2014 at 10:28 PM, Arjun <arjun@socialtwist.com> wrote:

> i changed the time to 60 seconds even now i see the same result. The
> Consumer is not consuming the messages.
>
> Thanks
> Arjun Narasimha Kota
>
>
> On Friday 11 April 2014 10:36 AM, Arjun wrote:
>
>> yup i will change the value and recheck. Thanks for the help.
>>
>> thanks
>> Arjun Narasimha Kota
>>
>> On Friday 11 April 2014 10:28 AM, Guozhang Wang wrote:
>>
>>> What I tried to say is that it may be caused by your
>>> "fetch.wait.max.ms"="180000"
>>> too large. Try a small value and see if that helps.
>>>
>>>
>>> On Thu, Apr 10, 2014 at 9:44 PM, Arjun <arjun@socialtwist.com> wrote:
>>>
>>>  Hi,
>>>>
>>>> I could not see any out of memory exceptions in the broker logs. One
>>>> thing
>>>> i can see is i  may have configured consumer poorly. If its not too
>>>> much to
>>>> ask can u let me know the changes i have to do for over coming this
>>>> problem.
>>>>
>>>> Thanks
>>>> Arjun Narasimha Kota
>>>>
>>>>
>>>> On Friday 11 April 2014 10:04 AM, Guozhang Wang wrote:
>>>>
>>>>  Hi Ajrun,
>>>>>
>>>>> It seems to be the cause:
>>>>>
>>>>> https://issues.apache.org/jira/browse/KAFKA-1016
>>>>>
>>>>> Guozhang
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 10, 2014 at 9:21 PM, Arjun <arjun@socialtwist.com>
wrote:
>>>>>
>>>>>   I hope this one would give u  a better idea.
>>>>>
>>>>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
>>>>>> group1
>>>>>> --zkconnect zkhost:port --topic testtopic
>>>>>> Group           Topic                          Pid Offset logSize
>>>>>> Lag             Owner
>>>>>> group1          testtopic    0   253             253 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>> group1          testtopic    1   267             267 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>> group1          testtopic    2   254             254 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>> group1          testtopic    3   265             265 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>> group1          testtopic    4   261             261 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>> group1          testtopic    5   294             294 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>> group1          testtopic    6   248             248 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>> group1          testtopic    7   271             271 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>> group1          testtopic    8   240             240 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>> group1          testtopic    9   261             261 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>> group1          testtopic    10  290             290 0
>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>> group1          testtopic    11  250             251 1
>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>>
>>>>>> If you see the output, in the last line the lag is 1 for that
>>>>>> partition.
>>>>>> I
>>>>>> just send one message. This topic is not new as you see there are
lot
>>>>>> of
>>>>>> messages which have accumlated from yesterday. This one message will
>>>>>> not
>>>>>> be
>>>>>> consumed by consumer what so ever. But if i send some 10 messages
then
>>>>>> all
>>>>>> the messages are consumed.
>>>>>>
>>>>>> Please let me know if i have to change any consumer properties.
>>>>>>
>>>>>> My consumer properties are :
>>>>>> "fetch.wait.max.ms"="180000"
>>>>>> "fetch.min.bytes" = "1"
>>>>>> "auto.offset.reset" = "smallest"
>>>>>> "auto.commit.enable"=  "false"
>>>>>> "fetch.message.max.bytes" = "1048576"
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Arjun Narasimha Kota
>>>>>> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
>>>>>>
>>>>>>   The consumer uses do specific topics.
>>>>>>
>>>>>>> On Apr 11, 2014 6:23 AM, "Arjun Kota" <arjun@socialtwist.com<mailto:
>>>>>>> arjun@socialtwist.com>> wrote:
>>>>>>>
>>>>>>>       Yes the message shows up on the server.
>>>>>>>
>>>>>>>       On Apr 11, 2014 12:07 AM, "Guozhang Wang" <wangguoz@gmail.com
>>>>>>>       <mailto:wangguoz@gmail.com>> wrote:
>>>>>>>
>>>>>>>           Hi Arjun,
>>>>>>>
>>>>>>>           If you only send one message, does that message show
up on
>>>>>>> the
>>>>>>>           server? Does
>>>>>>>           you consumer use wildcard topics or specific topics?
>>>>>>>
>>>>>>>           Guozhang
>>>>>>>
>>>>>>>
>>>>>>>           On Thu, Apr 10, 2014 at 9:20 AM, Arjun <
>>>>>>> arjun@socialtwist.com
>>>>>>>           <mailto:arjun@socialtwist.com>> wrote:
>>>>>>>
>>>>>>>           > But  we have auto offset reset to smallest not
largest,
>>>>>>> even
>>>>>>>           then this
>>>>>>>           > issue arises? If so is there any work around?
>>>>>>>           >
>>>>>>>           > Thanks
>>>>>>>           > Arjun NArasimha Kota
>>>>>>>           >
>>>>>>>           >
>>>>>>>           > On Thursday 10 April 2014 09:39 PM, Guozhang Wang
wrote:
>>>>>>>           >
>>>>>>>           >> It could be https://issues.apache.org/
>>>>>>> jira/browse/KAFKA-1006.
>>>>>>>           >>
>>>>>>>           >> Guozhang
>>>>>>>           >>
>>>>>>>           >>
>>>>>>>           >> On Thu, Apr 10, 2014 at 8:50 AM, Arjun
>>>>>>>           <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>
>>>>>>> wrote:
>>>>>>>           >>
>>>>>>>           >>  its auto created
>>>>>>>           >>> but even after topic creation this is
the scenario
>>>>>>>           >>>
>>>>>>>           >>> Arjun
>>>>>>>           >>>
>>>>>>>           >>> On Thursday 10 April 2014 08:41 PM, Guozhang
Wang
>>>>>>> wrote:
>>>>>>>           >>>
>>>>>>>           >>>  Hi Arjun,
>>>>>>>           >>>>
>>>>>>>           >>>> Did you manually create the topic
or use
>>>>>>> auto.topic.creation?
>>>>>>>           >>>>
>>>>>>>           >>>> Guozhang
>>>>>>>           >>>>
>>>>>>>           >>>>
>>>>>>>           >>>> On Thu, Apr 10, 2014 at 7:39 AM, Arjun
>>>>>>>           <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>
>>>>>>> wrote:
>>>>>>>           >>>>
>>>>>>>           >>>>   Hi,
>>>>>>>           >>>>
>>>>>>>           >>>>> We have 3 node kafka 0.8 setup
with zookeepers
>>>>>>> ensemble.
>>>>>>>           We use high
>>>>>>>           >>>>> level
>>>>>>>           >>>>> consumer with auto commit offset
false. I am facing
>>>>>>> some
>>>>>>>           peculiar
>>>>>>>           >>>>> problem
>>>>>>>           >>>>> with kafka. When i send some 10-20
messages or so the
>>>>>>>           consumer starts
>>>>>>>           >>>>> to
>>>>>>>           >>>>> consume the messages. But if 
i send only one
>>>>>>> message to
>>>>>>>           kafka, then
>>>>>>>           >>>>> even
>>>>>>>           >>>>> though consumer is active it is
not trying to fetch
>>>>>>> the
>>>>>>>           message. There
>>>>>>>           >>>>> is
>>>>>>>           >>>>> nothing in logs, just the messages
are being fetched
>>>>>>> by
>>>>>>>           the kafka
>>>>>>>           >>>>> consumer.
>>>>>>>           >>>>> The messages are there in the
Kafka server. Can some
>>>>>>> one
>>>>>>>           let me know
>>>>>>>           >>>>> where
>>>>>>>           >>>>> i am doing wrong.
>>>>>>>           >>>>>
>>>>>>>           >>>>>
>>>>>>>           >>>>> Thanks
>>>>>>>           >>>>> Arjun Narasimha Kota
>>>>>>>           >>>>>
>>>>>>>           >>>>>
>>>>>>>           >>>>>
>>>>>>>           >>>>
>>>>>>>           >>
>>>>>>>           >
>>>>>>>
>>>>>>>
>>>>>>>           --
>>>>>>>           -- Guozhang
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message