kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun <ar...@socialtwist.com>
Subject Re: consumer not consuming messages
Date Fri, 11 Apr 2014 05:28:46 GMT
i changed the time to 60 seconds even now i see the same result. The 
Consumer is not consuming the messages.

Thanks
Arjun Narasimha Kota

On Friday 11 April 2014 10:36 AM, Arjun wrote:
> yup i will change the value and recheck. Thanks for the help.
>
> thanks
> Arjun Narasimha Kota
>
> On Friday 11 April 2014 10:28 AM, Guozhang Wang wrote:
>> What I tried to say is that it may be caused by your
>> "fetch.wait.max.ms"="180000"
>> too large. Try a small value and see if that helps.
>>
>>
>> On Thu, Apr 10, 2014 at 9:44 PM, Arjun <arjun@socialtwist.com> wrote:
>>
>>> Hi,
>>>
>>> I could not see any out of memory exceptions in the broker logs. One 
>>> thing
>>> i can see is i  may have configured consumer poorly. If its not too 
>>> much to
>>> ask can u let me know the changes i have to do for over coming this 
>>> problem.
>>>
>>> Thanks
>>> Arjun Narasimha Kota
>>>
>>>
>>> On Friday 11 April 2014 10:04 AM, Guozhang Wang wrote:
>>>
>>>> Hi Ajrun,
>>>>
>>>> It seems to be the cause:
>>>>
>>>> https://issues.apache.org/jira/browse/KAFKA-1016
>>>>
>>>> Guozhang
>>>>
>>>>
>>>>
>>>> On Thu, Apr 10, 2014 at 9:21 PM, Arjun <arjun@socialtwist.com> wrote:
>>>>
>>>>   I hope this one would give u  a better idea.
>>>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group 
>>>>> group1
>>>>> --zkconnect zkhost:port --topic testtopic
>>>>> Group           Topic                          Pid Offset logSize
>>>>> Lag             Owner
>>>>> group1          testtopic    0   253             253 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>> group1          testtopic    1   267             267 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>> group1          testtopic    2   254             254 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>> group1          testtopic    3   265             265 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-0
>>>>> group1          testtopic    4   261             261 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>> group1          testtopic    5   294             294 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>> group1          testtopic    6   248             248 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>> group1          testtopic    7   271             271 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-1
>>>>> group1          testtopic    8   240             240 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>> group1          testtopic    9   261             261 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>> group1          testtopic    10  290             290 0
>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>> group1          testtopic    11  250             251 1
>>>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>>>
>>>>> If you see the output, in the last line the lag is 1 for that 
>>>>> partition.
>>>>> I
>>>>> just send one message. This topic is not new as you see there are 
>>>>> lot of
>>>>> messages which have accumlated from yesterday. This one message 
>>>>> will not
>>>>> be
>>>>> consumed by consumer what so ever. But if i send some 10 messages 
>>>>> then
>>>>> all
>>>>> the messages are consumed.
>>>>>
>>>>> Please let me know if i have to change any consumer properties.
>>>>>
>>>>> My consumer properties are :
>>>>> "fetch.wait.max.ms"="180000"
>>>>> "fetch.min.bytes" = "1"
>>>>> "auto.offset.reset" = "smallest"
>>>>> "auto.commit.enable"=  "false"
>>>>> "fetch.message.max.bytes" = "1048576"
>>>>>
>>>>>
>>>>> Thanks
>>>>> Arjun Narasimha Kota
>>>>> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
>>>>>
>>>>>   The consumer uses do specific topics.
>>>>>> On Apr 11, 2014 6:23 AM, "Arjun Kota" <arjun@socialtwist.com 
>>>>>> <mailto:
>>>>>> arjun@socialtwist.com>> wrote:
>>>>>>
>>>>>>       Yes the message shows up on the server.
>>>>>>
>>>>>>       On Apr 11, 2014 12:07 AM, "Guozhang Wang" <wangguoz@gmail.com
>>>>>>       <mailto:wangguoz@gmail.com>> wrote:
>>>>>>
>>>>>>           Hi Arjun,
>>>>>>
>>>>>>           If you only send one message, does that message show up

>>>>>> on the
>>>>>>           server? Does
>>>>>>           you consumer use wildcard topics or specific topics?
>>>>>>
>>>>>>           Guozhang
>>>>>>
>>>>>>
>>>>>>           On Thu, Apr 10, 2014 at 9:20 AM, Arjun 
>>>>>> <arjun@socialtwist.com
>>>>>>           <mailto:arjun@socialtwist.com>> wrote:
>>>>>>
>>>>>>           > But  we have auto offset reset to smallest not 
>>>>>> largest, even
>>>>>>           then this
>>>>>>           > issue arises? If so is there any work around?
>>>>>>           >
>>>>>>           > Thanks
>>>>>>           > Arjun NArasimha Kota
>>>>>>           >
>>>>>>           >
>>>>>>           > On Thursday 10 April 2014 09:39 PM, Guozhang Wang
wrote:
>>>>>>           >
>>>>>>           >> It could be https://issues.apache.org/
>>>>>> jira/browse/KAFKA-1006.
>>>>>>           >>
>>>>>>           >> Guozhang
>>>>>>           >>
>>>>>>           >>
>>>>>>           >> On Thu, Apr 10, 2014 at 8:50 AM, Arjun
>>>>>>           <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>

>>>>>> wrote:
>>>>>>           >>
>>>>>>           >>  its auto created
>>>>>>           >>> but even after topic creation this is the
scenario
>>>>>>           >>>
>>>>>>           >>> Arjun
>>>>>>           >>>
>>>>>>           >>> On Thursday 10 April 2014 08:41 PM, Guozhang
Wang 
>>>>>> wrote:
>>>>>>           >>>
>>>>>>           >>>  Hi Arjun,
>>>>>>           >>>>
>>>>>>           >>>> Did you manually create the topic or use
>>>>>> auto.topic.creation?
>>>>>>           >>>>
>>>>>>           >>>> Guozhang
>>>>>>           >>>>
>>>>>>           >>>>
>>>>>>           >>>> On Thu, Apr 10, 2014 at 7:39 AM, Arjun
>>>>>>           <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>

>>>>>> wrote:
>>>>>>           >>>>
>>>>>>           >>>>   Hi,
>>>>>>           >>>>
>>>>>>           >>>>> We have 3 node kafka 0.8 setup with
zookeepers 
>>>>>> ensemble.
>>>>>>           We use high
>>>>>>           >>>>> level
>>>>>>           >>>>> consumer with auto commit offset false.
I am 
>>>>>> facing some
>>>>>>           peculiar
>>>>>>           >>>>> problem
>>>>>>           >>>>> with kafka. When i send some 10-20
messages or so 
>>>>>> the
>>>>>>           consumer starts
>>>>>>           >>>>> to
>>>>>>           >>>>> consume the messages. But if  i send
only one 
>>>>>> message to
>>>>>>           kafka, then
>>>>>>           >>>>> even
>>>>>>           >>>>> though consumer is active it is not
trying to 
>>>>>> fetch the
>>>>>>           message. There
>>>>>>           >>>>> is
>>>>>>           >>>>> nothing in logs, just the messages
are being 
>>>>>> fetched by
>>>>>>           the kafka
>>>>>>           >>>>> consumer.
>>>>>>           >>>>> The messages are there in the Kafka
server. Can 
>>>>>> some one
>>>>>>           let me know
>>>>>>           >>>>> where
>>>>>>           >>>>> i am doing wrong.
>>>>>>           >>>>>
>>>>>>           >>>>>
>>>>>>           >>>>> Thanks
>>>>>>           >>>>> Arjun Narasimha Kota
>>>>>>           >>>>>
>>>>>>           >>>>>
>>>>>>           >>>>>
>>>>>>           >>>>
>>>>>>           >>
>>>>>>           >
>>>>>>
>>>>>>
>>>>>>           --
>>>>>>           -- Guozhang
>>>>>>
>>>>>>
>>>>>>
>>
>


Mime
View raw message