kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: consumer not consuming messages
Date Mon, 14 Apr 2014 15:16:48 GMT
"Rebalancing attempt failed" indicates the rebalancing failed. I added some
notes in the last item in
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why
?

Thanks,

Jun


On Fri, Apr 11, 2014 at 11:23 PM, Arjun <arjun@socialtwist.com> wrote:

> Even after changing the  fetch wait max ms the same thing is repeting just
> that some partitions have the owners now, i mean
>
>
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
> --zkconnect zkhost:zkport --topic testtopic
>
> Group           Topic                          Pid Offset logSize
> Lag             Owner
> group1          testtopic    0   253             253 0
> group1_xxxx-1397216047177-6f419d28-0
> group1          testtopic    1   268             268 0
> group1_xxxx-1397216047177-6f419d28-1
> group1          testtopic    2   258             258 0
> group1_xxxx-1397216047177-6f419d28-2
> group1          testtopic    3   265             265 0               none
> group1          testtopic    4   262             262 0               none
> group1          testtopic    5   296             296 0               none
> group1          testtopic    6   249             249 0
> group1_xxxx1-1397219018735-a2cf60df-0
> group1          testtopic    7   272             272 0
> group1_xxxx1-1397219018735-a2cf60df-1
> group1          testtopic    8   242             242 0
> group1_xxxx1-1397219018735-a2cf60df-2
> group1          testtopic    9   263             263 0
> group1_xxxx1-1397279906839-12366659-0
> group1          testtopic    10  294             294 0
> group1_xxxx1-1397279906839-12366659-1
> group1          testtopic    11  254             254 0
> group1_xxxx1-1397279906839-12366659-2
>
> and i dont see any exception after
>
> "end rebalancing consumer group1_ip-XXXX-82095a28 try #9"
>
> i just see the below log
>
> 12 Apr 2014 00:18:58,987 INFO  [Thread-321] [kafka.consumer.ZookeeperConsumerConnector]
> [group1_ip-xxxx-82095a28], Rebalancing attempt failed. Clearing the cache
> before the next rebalancing operation is triggered
> 12 Apr 2014 00:18:58,987 INFO  [Thread-321] [kafka.consumer.ConsumerFetcherManager]
> [ConsumerFetcherManager-1397279912725] Stopping leader finder thread
> 12 Apr 2014 00:18:58,987 INFO  [Thread-321] [kafka.consumer.ConsumerFetcherManager]
> [ConsumerFetcherManager-1397279912725] Stopping all fetchers
> 12 Apr 2014 00:18:58,987 INFO  [Thread-321] [kafka.consumer.ConsumerFetcherManager]
> [ConsumerFetcherManager-1397279912725] All connections stopped
> 12 Apr 2014 00:18:58,987 INFO  [Thread-321] [kafka.consumer.ZookeeperConsumerConnector]
> [group1_xxxx-82095a28], Cleared all relevant queues for this fetcher
> 12 Apr 2014 00:18:58,988 DEBUG [Thread-321] [kafka.consumer.ConsumerIterator]
> Clearing the current data chunk for this consumer iterator
> 12 Apr 2014 00:18:58,988 DEBUG [Thread-321] [kafka.consumer.ConsumerIterator]
> Clearing the current data chunk for this consumer iterator
> 12 Apr 2014 00:18:58,988 DEBUG [Thread-321] [kafka.consumer.ConsumerIterator]
> Clearing the current data chunk for this consumer iterator
> 12 Apr 2014 00:18:58,988 INFO  [Thread-321] [kafka.consumer.ZookeeperConsumerConnector]
> [group1_xxxx-82095a28], Cleared the data chunks in all the consumer message
> iterators
> 12 Apr 2014 00:18:58,988 INFO  [Thread-321] [kafka.consumer.ZookeeperConsumerConnector]
> [group1_xxxx-82095a28], Committing all offsets after clearing the fetcher
> queues
>
> after this kafka consumer will not log a thing.Not even the exception.(I
> have put my consumer log level to debug)
>
>
>
> Thanks
> Arjun Narasimha Kota
>
>
>
> On Saturday 12 April 2014 08:41 AM, Jun Rao wrote:
>
>> Console consumer also uses the high level consumer. Could you try setting
>> fetch.wait.max.ms to 100ms?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Fri, Apr 11, 2014 at 9:56 AM, Arjun Kota <arjun@socialtwist.com>
>> wrote:
>>
>>  Console consumer works fine. Its the high level java consumer which is
>>> giving this problem.
>>>
>>> Thanks
>>> Arjun narasimha kota
>>> On Apr 11, 2014 8:42 PM, "Jun Rao" <junrao@gmail.com> wrote:
>>>
>>>  We may have a bug that doesn't observe etch.min.bytes accurately. So a
>>>> lower fetch.wait.max.ms will improve consumer latency.
>>>>
>>>> Could you run a console consumer and see if you have the same issue?
>>>> That
>>>> will tell us if this is a server side issue or an issue just in your
>>>> consumer.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Thu, Apr 10, 2014 at 10:28 PM, Arjun <arjun@socialtwist.com> wrote:
>>>>
>>>>  i changed the time to 60 seconds even now i see the same result. The
>>>>> Consumer is not consuming the messages.
>>>>>
>>>>> Thanks
>>>>> Arjun Narasimha Kota
>>>>>
>>>>>
>>>>> On Friday 11 April 2014 10:36 AM, Arjun wrote:
>>>>>
>>>>>  yup i will change the value and recheck. Thanks for the help.
>>>>>>
>>>>>> thanks
>>>>>> Arjun Narasimha Kota
>>>>>>
>>>>>> On Friday 11 April 2014 10:28 AM, Guozhang Wang wrote:
>>>>>>
>>>>>>  What I tried to say is that it may be caused by your
>>>>>>> "fetch.wait.max.ms"="180000"
>>>>>>> too large. Try a small value and see if that helps.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 10, 2014 at 9:44 PM, Arjun <arjun@socialtwist.com>
>>>>>>>
>>>>>> wrote:
>>>
>>>>   Hi,
>>>>>>>
>>>>>>>> I could not see any out of memory exceptions in the broker
logs. One
>>>>>>>> thing
>>>>>>>> i can see is i  may have configured consumer poorly. If its
not too
>>>>>>>> much to
>>>>>>>> ask can u let me know the changes i have to do for over coming
this
>>>>>>>> problem.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Arjun Narasimha Kota
>>>>>>>>
>>>>>>>>
>>>>>>>> On Friday 11 April 2014 10:04 AM, Guozhang Wang wrote:
>>>>>>>>
>>>>>>>>   Hi Ajrun,
>>>>>>>>
>>>>>>>>> It seems to be the cause:
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-1016
>>>>>>>>>
>>>>>>>>> Guozhang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Apr 10, 2014 at 9:21 PM, Arjun <arjun@socialtwist.com>
>>>>>>>>>
>>>>>>>> wrote:
>>>>
>>>>>    I hope this one would give u  a better idea.
>>>>>>>>>
>>>>>>>>>  bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
--group
>>>>>>>>>> group1
>>>>>>>>>> --zkconnect zkhost:port --topic testtopic
>>>>>>>>>> Group           Topic                          Pid
Offset logSize
>>>>>>>>>> Lag             Owner
>>>>>>>>>> group1          testtopic    0   253            
253 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>>>>>> group1          testtopic    1   267            
267 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>>>>>> group1          testtopic    2   254            
254 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>>>>>> group1          testtopic    3   265            
265 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>>>>>>> group1          testtopic    4   261            
261 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>>>>>> group1          testtopic    5   294            
294 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>>>>>> group1          testtopic    6   248            
248 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>>>>>> group1          testtopic    7   271            
271 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>>>>>>> group1          testtopic    8   240            
240 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>>>>>> group1          testtopic    9   261            
261 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>>>>>> group1          testtopic    10  290            
290 0
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>>>>>> group1          testtopic    11  250            
251 1
>>>>>>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>>>>>>
>>>>>>>>>> If you see the output, in the last line the lag is
1 for that
>>>>>>>>>> partition.
>>>>>>>>>> I
>>>>>>>>>> just send one message. This topic is not new as you
see there are
>>>>>>>>>>
>>>>>>>>> lot
>>>>
>>>>> of
>>>>>>>>>> messages which have accumlated from yesterday. This
one message
>>>>>>>>>>
>>>>>>>>> will
>>>
>>>> not
>>>>>>>>>> be
>>>>>>>>>> consumed by consumer what so ever. But if i send
some 10 messages
>>>>>>>>>>
>>>>>>>>> then
>>>>
>>>>> all
>>>>>>>>>> the messages are consumed.
>>>>>>>>>>
>>>>>>>>>> Please let me know if i have to change any consumer
properties.
>>>>>>>>>>
>>>>>>>>>> My consumer properties are :
>>>>>>>>>> "fetch.wait.max.ms"="180000"
>>>>>>>>>> "fetch.min.bytes" = "1"
>>>>>>>>>> "auto.offset.reset" = "smallest"
>>>>>>>>>> "auto.commit.enable"=  "false"
>>>>>>>>>> "fetch.message.max.bytes" = "1048576"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Arjun Narasimha Kota
>>>>>>>>>> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
>>>>>>>>>>
>>>>>>>>>>    The consumer uses do specific topics.
>>>>>>>>>>
>>>>>>>>>>  On Apr 11, 2014 6:23 AM, "Arjun Kota" <arjun@socialtwist.com
>>>>>>>>>>>
>>>>>>>>>> <mailto:
>>>>
>>>>> arjun@socialtwist.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>        Yes the message shows up on the server.
>>>>>>>>>>>
>>>>>>>>>>>        On Apr 11, 2014 12:07 AM, "Guozhang Wang"
<
>>>>>>>>>>>
>>>>>>>>>> wangguoz@gmail.com
>>>>
>>>>>        <mailto:wangguoz@gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>            Hi Arjun,
>>>>>>>>>>>
>>>>>>>>>>>            If you only send one message, does
that message show
>>>>>>>>>>> up
>>>>>>>>>>>
>>>>>>>>>> on
>>>>
>>>>> the
>>>>>>>>>>>            server? Does
>>>>>>>>>>>            you consumer use wildcard topics or
specific topics?
>>>>>>>>>>>
>>>>>>>>>>>            Guozhang
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>            On Thu, Apr 10, 2014 at 9:20 AM, Arjun
<
>>>>>>>>>>> arjun@socialtwist.com
>>>>>>>>>>>            <mailto:arjun@socialtwist.com>>
wrote:
>>>>>>>>>>>
>>>>>>>>>>>            > But  we have auto offset reset
to smallest not
>>>>>>>>>>>
>>>>>>>>>> largest,
>>>
>>>> even
>>>>>>>>>>>            then this
>>>>>>>>>>>            > issue arises? If so is there
any work around?
>>>>>>>>>>>            >
>>>>>>>>>>>            > Thanks
>>>>>>>>>>>            > Arjun NArasimha Kota
>>>>>>>>>>>            >
>>>>>>>>>>>            >
>>>>>>>>>>>            > On Thursday 10 April 2014 09:39
PM, Guozhang Wang
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>
>>>>>            >
>>>>>>>>>>>            >> It could be https://issues.apache.org/
>>>>>>>>>>> jira/browse/KAFKA-1006.
>>>>>>>>>>>            >>
>>>>>>>>>>>            >> Guozhang
>>>>>>>>>>>            >>
>>>>>>>>>>>            >>
>>>>>>>>>>>            >> On Thu, Apr 10, 2014 at 8:50
AM, Arjun
>>>>>>>>>>>            <arjun@socialtwist.com <mailto:arjun@socialtwist.com>
>>>>>>>>>>> >
>>>>>>>>>>> wrote:
>>>>>>>>>>>            >>
>>>>>>>>>>>            >>  its auto created
>>>>>>>>>>>            >>> but even after topic
creation this is the scenario
>>>>>>>>>>>            >>>
>>>>>>>>>>>            >>> Arjun
>>>>>>>>>>>            >>>
>>>>>>>>>>>            >>> On Thursday 10 April
2014 08:41 PM, Guozhang Wang
>>>>>>>>>>> wrote:
>>>>>>>>>>>            >>>
>>>>>>>>>>>            >>>  Hi Arjun,
>>>>>>>>>>>            >>>>
>>>>>>>>>>>            >>>> Did you manually
create the topic or use
>>>>>>>>>>> auto.topic.creation?
>>>>>>>>>>>            >>>>
>>>>>>>>>>>            >>>> Guozhang
>>>>>>>>>>>            >>>>
>>>>>>>>>>>            >>>>
>>>>>>>>>>>            >>>> On Thu, Apr 10, 2014
at 7:39 AM, Arjun
>>>>>>>>>>>            <arjun@socialtwist.com <mailto:arjun@socialtwist.com>
>>>>>>>>>>> >
>>>>>>>>>>> wrote:
>>>>>>>>>>>            >>>>
>>>>>>>>>>>            >>>>   Hi,
>>>>>>>>>>>            >>>>
>>>>>>>>>>>            >>>>> We have 3 node
kafka 0.8 setup with zookeepers
>>>>>>>>>>> ensemble.
>>>>>>>>>>>            We use high
>>>>>>>>>>>            >>>>> level
>>>>>>>>>>>            >>>>> consumer with
auto commit offset false. I am
>>>>>>>>>>>
>>>>>>>>>> facing
>>>
>>>> some
>>>>>>>>>>>            peculiar
>>>>>>>>>>>            >>>>> problem
>>>>>>>>>>>            >>>>> with kafka. When
i send some 10-20 messages or
>>>>>>>>>>> so
>>>>>>>>>>>
>>>>>>>>>> the
>>>>
>>>>>            consumer starts
>>>>>>>>>>>            >>>>> to
>>>>>>>>>>>            >>>>> consume the messages.
But if  i send only one
>>>>>>>>>>> message to
>>>>>>>>>>>            kafka, then
>>>>>>>>>>>            >>>>> even
>>>>>>>>>>>            >>>>> though consumer
is active it is not trying to
>>>>>>>>>>>
>>>>>>>>>> fetch
>>>
>>>> the
>>>>>>>>>>>            message. There
>>>>>>>>>>>            >>>>> is
>>>>>>>>>>>            >>>>> nothing in logs,
just the messages are being
>>>>>>>>>>>
>>>>>>>>>> fetched
>>>>
>>>>> by
>>>>>>>>>>>            the kafka
>>>>>>>>>>>            >>>>> consumer.
>>>>>>>>>>>            >>>>> The messages
are there in the Kafka server. Can
>>>>>>>>>>>
>>>>>>>>>> some
>>>>
>>>>> one
>>>>>>>>>>>            let me know
>>>>>>>>>>>            >>>>> where
>>>>>>>>>>>            >>>>> i am doing wrong.
>>>>>>>>>>>            >>>>>
>>>>>>>>>>>            >>>>>
>>>>>>>>>>>            >>>>> Thanks
>>>>>>>>>>>            >>>>> Arjun Narasimha
Kota
>>>>>>>>>>>            >>>>>
>>>>>>>>>>>            >>>>>
>>>>>>>>>>>            >>>>>
>>>>>>>>>>>            >>>>
>>>>>>>>>>>            >>
>>>>>>>>>>>            >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>            --
>>>>>>>>>>>            -- Guozhang
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message