kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: consumer not consuming messages
Date Fri, 11 Apr 2014 04:58:26 GMT
What I tried to say is that it may be caused by your
"fetch.wait.max.ms"="180000"
too large. Try a small value and see if that helps.


On Thu, Apr 10, 2014 at 9:44 PM, Arjun <arjun@socialtwist.com> wrote:

> Hi,
>
> I could not see any out of memory exceptions in the broker logs. One thing
> i can see is i  may have configured consumer poorly. If its not too much to
> ask can u let me know the changes i have to do for over coming this problem.
>
> Thanks
> Arjun Narasimha Kota
>
>
> On Friday 11 April 2014 10:04 AM, Guozhang Wang wrote:
>
>> Hi Ajrun,
>>
>> It seems to be the cause:
>>
>> https://issues.apache.org/jira/browse/KAFKA-1016
>>
>> Guozhang
>>
>>
>>
>> On Thu, Apr 10, 2014 at 9:21 PM, Arjun <arjun@socialtwist.com> wrote:
>>
>>  I hope this one would give u  a better idea.
>>>
>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
>>> --zkconnect zkhost:port --topic testtopic
>>> Group           Topic                          Pid Offset logSize
>>> Lag             Owner
>>> group1          testtopic    0   253             253 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    1   267             267 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    2   254             254 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    3   265             265 0
>>> group1_ip-xx-1397188061429-b5ff1205-0
>>> group1          testtopic    4   261             261 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    5   294             294 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    6   248             248 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    7   271             271 0
>>> group1_ip-xx-1397188061429-b5ff1205-1
>>> group1          testtopic    8   240             240 0
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>> group1          testtopic    9   261             261 0
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>> group1          testtopic    10  290             290 0
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>> group1          testtopic    11  250             251 1
>>> group1_ip-xx-1397188061429-b5ff1205-2
>>>
>>> If you see the output, in the last line the lag is 1 for that partition.
>>> I
>>> just send one message. This topic is not new as you see there are lot of
>>> messages which have accumlated from yesterday. This one message will not
>>> be
>>> consumed by consumer what so ever. But if i send some 10 messages then
>>> all
>>> the messages are consumed.
>>>
>>> Please let me know if i have to change any consumer properties.
>>>
>>> My consumer properties are :
>>> "fetch.wait.max.ms"="180000"
>>> "fetch.min.bytes" = "1"
>>> "auto.offset.reset" = "smallest"
>>> "auto.commit.enable"=  "false"
>>> "fetch.message.max.bytes" = "1048576"
>>>
>>>
>>> Thanks
>>> Arjun Narasimha Kota
>>> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
>>>
>>>  The consumer uses do specific topics.
>>>>
>>>> On Apr 11, 2014 6:23 AM, "Arjun Kota" <arjun@socialtwist.com <mailto:
>>>> arjun@socialtwist.com>> wrote:
>>>>
>>>>      Yes the message shows up on the server.
>>>>
>>>>      On Apr 11, 2014 12:07 AM, "Guozhang Wang" <wangguoz@gmail.com
>>>>      <mailto:wangguoz@gmail.com>> wrote:
>>>>
>>>>          Hi Arjun,
>>>>
>>>>          If you only send one message, does that message show up on the
>>>>          server? Does
>>>>          you consumer use wildcard topics or specific topics?
>>>>
>>>>          Guozhang
>>>>
>>>>
>>>>          On Thu, Apr 10, 2014 at 9:20 AM, Arjun <arjun@socialtwist.com
>>>>          <mailto:arjun@socialtwist.com>> wrote:
>>>>
>>>>          > But  we have auto offset reset to smallest not largest, even
>>>>          then this
>>>>          > issue arises? If so is there any work around?
>>>>          >
>>>>          > Thanks
>>>>          > Arjun NArasimha Kota
>>>>          >
>>>>          >
>>>>          > On Thursday 10 April 2014 09:39 PM, Guozhang Wang wrote:
>>>>          >
>>>>          >> It could be https://issues.apache.org/
>>>> jira/browse/KAFKA-1006.
>>>>          >>
>>>>          >> Guozhang
>>>>          >>
>>>>          >>
>>>>          >> On Thu, Apr 10, 2014 at 8:50 AM, Arjun
>>>>          <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>
wrote:
>>>>          >>
>>>>          >>  its auto created
>>>>          >>> but even after topic creation this is the scenario
>>>>          >>>
>>>>          >>> Arjun
>>>>          >>>
>>>>          >>> On Thursday 10 April 2014 08:41 PM, Guozhang Wang wrote:
>>>>          >>>
>>>>          >>>  Hi Arjun,
>>>>          >>>>
>>>>          >>>> Did you manually create the topic or use
>>>> auto.topic.creation?
>>>>          >>>>
>>>>          >>>> Guozhang
>>>>          >>>>
>>>>          >>>>
>>>>          >>>> On Thu, Apr 10, 2014 at 7:39 AM, Arjun
>>>>          <arjun@socialtwist.com <mailto:arjun@socialtwist.com>>
wrote:
>>>>          >>>>
>>>>          >>>>   Hi,
>>>>          >>>>
>>>>          >>>>> We have 3 node kafka 0.8 setup with zookeepers
ensemble.
>>>>          We use high
>>>>          >>>>> level
>>>>          >>>>> consumer with auto commit offset false. I am
facing some
>>>>          peculiar
>>>>          >>>>> problem
>>>>          >>>>> with kafka. When i send some 10-20 messages
or so the
>>>>          consumer starts
>>>>          >>>>> to
>>>>          >>>>> consume the messages. But if  i send only one
message to
>>>>          kafka, then
>>>>          >>>>> even
>>>>          >>>>> though consumer is active it is not trying
to fetch the
>>>>          message. There
>>>>          >>>>> is
>>>>          >>>>> nothing in logs, just the messages are being
fetched by
>>>>          the kafka
>>>>          >>>>> consumer.
>>>>          >>>>> The messages are there in the Kafka server.
Can some one
>>>>          let me know
>>>>          >>>>> where
>>>>          >>>>> i am doing wrong.
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>> Thanks
>>>>          >>>>> Arjun Narasimha Kota
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>>
>>>>          >>>>
>>>>          >>
>>>>          >
>>>>
>>>>
>>>>          --
>>>>          -- Guozhang
>>>>
>>>>
>>>>
>>
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message