kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun <ar...@socialtwist.com>
Subject Re: consumer not consuming messages
Date Fri, 11 Apr 2014 05:04:41 GMT
Hi,

 From my understanding, the fetch wait max time is the maximum time the 
consumer waits if there are no messages in the broker. If there are 
messages in the broker, it just gets all the messages from the broker.Is 
my understanding wrong?

thanks
Arjun Narasimha Kota

On Friday 11 April 2014 10:14 AM, Arjun wrote:
> Hi,
>
> I could not see any out of memory exceptions in the broker logs. One 
> thing i can see is i  may have configured consumer poorly. If its not 
> too much to ask can u let me know the changes i have to do for over 
> coming this problem.
>
> Thanks
> Arjun Narasimha Kota
>
> On Friday 11 April 2014 10:04 AM, Guozhang Wang wrote:
>> Hi Ajrun,
>>
>> It seems to be the cause:
>>
>> https://issues.apache.org/jira/browse/KAFKA-1016
>>
>> Guozhang
>>
>>
>>
>> On Thu, Apr 10, 2014 at 9:21 PM, Arjun <arjun@socialtwist.com> wrote:
>>
>>> I hope this one would give u  a better idea.
>>>
>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
>>> --zkconnect zkhost:port --topic testtopic
>>> Group           Topic                          Pid Offset logSize
>>> Lag             Owner
>>> group1          testtopic    0   253             253 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    1   267             267 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    2   254             254 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    3   265             265 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    4   261             261 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    5   294             294 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    6   248             248 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    7   271             271 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    8   240             240 0
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>> group1          testtopic    9   261             261 0
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>> group1          testtopic    10  290             290 0
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>> group1          testtopic    11  250             251 1
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>
>>> If you see the output, in the last line the lag is 1 for that 
>>> partition. I
>>> just send one message. This topic is not new as you see there are 
>>> lot of
>>> messages which have accumlated from yesterday. This one message will 
>>> not be
>>> consumed by consumer what so ever. But if i send some 10 messages 
>>> then all
>>> the messages are consumed.
>>>
>>> Please let me know if i have to change any consumer properties.
>>>
>>> My consumer properties are :
>>> "fetch.wait.max.ms"="180000"
>>> "fetch.min.bytes" = "1"
>>> "auto.offset.reset" = "smallest"
>>> "auto.commit.enable"=  "false"
>>> "fetch.message.max.bytes" = "1048576"
>>>
>>>
>>> Thanks
>>> Arjun Narasimha Kota
>>> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
>>>
>>>> The consumer uses do specific topics.
>>>>
>>>> On Apr 11, 2014 6:23 AM, "Arjun Kota" <arjun@socialtwist.com <mailto:
>>>> arjun@socialtwist.com>> wrote:
>>>>
>>>>      Yes the message shows up on the server.
>>>>
>>>>      On Apr 11, 2014 12:07 AM, "Guozhang Wang" <wangguoz@gmail.com
>>>>      <mailto:wangguoz@gmail.com>> wrote:
>>>>
>>>>          Hi Arjun,
>>>>
>>>>          If you only send one message, does that message show up on 
>>>> the
>>>>          server? Does
>>>>          you consumer use wildcard topics or specific topics?
>>>>
>>>>          Guozhang
>>>>
>>>>
>>>>          On Thu, Apr 10, 2014 at 9:20 AM, Arjun <arjun@socialtwist.com
>>>>          <mailto:arjun@socialtwist.com>> wrote:
>>>>
>>>>          > But  we have auto offset reset to smallest not largest, 
>>>> even
>>>>          then this
>>>>          > issue arises? If so is there any work around?
>>>>          >
>>>>          > Thanks
>>>>          > Arjun NArasimha Kota
>>>>          >
>>>>          >
>>>>          > On Thursday 10 April 2014 09:39 PM, Guozhang Wang wrote:
>>>>          >
>>>>          >> It could be 
>>>> https://issues.apache.org/jira/browse/KAFKA-1006.
>>>>          >>
>>>>          >> Guozhang
>>>>          >>
>>>>          >>
>>>>          >> On Thu, Apr 10, 2014 at 8:50 AM, Arjun
>>>>          <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>
wrote:
>>>>          >>
>>>>          >>  its auto created
>>>>          >>> but even after topic creation this is the scenario
>>>>          >>>
>>>>          >>> Arjun
>>>>          >>>
>>>>          >>> On Thursday 10 April 2014 08:41 PM, Guozhang Wang wrote:
>>>>          >>>
>>>>          >>>  Hi Arjun,
>>>>          >>>>
>>>>          >>>> Did you manually create the topic or use 
>>>> auto.topic.creation?
>>>>          >>>>
>>>>          >>>> Guozhang
>>>>          >>>>
>>>>          >>>>
>>>>          >>>> On Thu, Apr 10, 2014 at 7:39 AM, Arjun
>>>>          <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>
wrote:
>>>>          >>>>
>>>>          >>>>   Hi,
>>>>          >>>>
>>>>          >>>>> We have 3 node kafka 0.8 setup with zookeepers

>>>> ensemble.
>>>>          We use high
>>>>          >>>>> level
>>>>          >>>>> consumer with auto commit offset false. I am
facing 
>>>> some
>>>>          peculiar
>>>>          >>>>> problem
>>>>          >>>>> with kafka. When i send some 10-20 messages
or so the
>>>>          consumer starts
>>>>          >>>>> to
>>>>          >>>>> consume the messages. But if i send only one
message to
>>>>          kafka, then
>>>>          >>>>> even
>>>>          >>>>> though consumer is active it is not trying
to fetch the
>>>>          message. There
>>>>          >>>>> is
>>>>          >>>>> nothing in logs, just the messages are being
fetched by
>>>>          the kafka
>>>>          >>>>> consumer.
>>>>          >>>>> The messages are there in the Kafka server.
Can some 
>>>> one
>>>>          let me know
>>>>          >>>>> where
>>>>          >>>>> i am doing wrong.
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>> Thanks
>>>>          >>>>> Arjun Narasimha Kota
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>
>>>>          >>
>>>>          >
>>>>
>>>>
>>>>          --
>>>>          -- Guozhang
>>>>
>>>>
>>
>


Mime
View raw message