kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun Kota <ar...@socialtwist.com>
Subject Re: consumer not consuming messages
Date Sat, 12 Apr 2014 03:35:53 GMT
Yup will try that
On Apr 12, 2014 8:42 AM, "Jun Rao" <junrao@gmail.com> wrote:

> Console consumer also uses the high level consumer. Could you try setting
> fetch.wait.max.ms to 100ms?
>
> Thanks,
>
> Jun
>
>
> On Fri, Apr 11, 2014 at 9:56 AM, Arjun Kota <arjun@socialtwist.com> wrote:
>
> > Console consumer works fine. Its the high level java consumer which is
> > giving this problem.
> >
> > Thanks
> > Arjun narasimha kota
> > On Apr 11, 2014 8:42 PM, "Jun Rao" <junrao@gmail.com> wrote:
> >
> > > We may have a bug that doesn't observe etch.min.bytes accurately. So a
> > > lower fetch.wait.max.ms will improve consumer latency.
> > >
> > > Could you run a console consumer and see if you have the same issue?
> That
> > > will tell us if this is a server side issue or an issue just in your
> > > consumer.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Apr 10, 2014 at 10:28 PM, Arjun <arjun@socialtwist.com> wrote:
> > >
> > > > i changed the time to 60 seconds even now i see the same result. The
> > > > Consumer is not consuming the messages.
> > > >
> > > > Thanks
> > > > Arjun Narasimha Kota
> > > >
> > > >
> > > > On Friday 11 April 2014 10:36 AM, Arjun wrote:
> > > >
> > > >> yup i will change the value and recheck. Thanks for the help.
> > > >>
> > > >> thanks
> > > >> Arjun Narasimha Kota
> > > >>
> > > >> On Friday 11 April 2014 10:28 AM, Guozhang Wang wrote:
> > > >>
> > > >>> What I tried to say is that it may be caused by your
> > > >>> "fetch.wait.max.ms"="180000"
> > > >>> too large. Try a small value and see if that helps.
> > > >>>
> > > >>>
> > > >>> On Thu, Apr 10, 2014 at 9:44 PM, Arjun <arjun@socialtwist.com>
> > wrote:
> > > >>>
> > > >>>  Hi,
> > > >>>>
> > > >>>> I could not see any out of memory exceptions in the broker
logs.
> One
> > > >>>> thing
> > > >>>> i can see is i  may have configured consumer poorly. If its
not
> too
> > > >>>> much to
> > > >>>> ask can u let me know the changes i have to do for over coming
> this
> > > >>>> problem.
> > > >>>>
> > > >>>> Thanks
> > > >>>> Arjun Narasimha Kota
> > > >>>>
> > > >>>>
> > > >>>> On Friday 11 April 2014 10:04 AM, Guozhang Wang wrote:
> > > >>>>
> > > >>>>  Hi Ajrun,
> > > >>>>>
> > > >>>>> It seems to be the cause:
> > > >>>>>
> > > >>>>> https://issues.apache.org/jira/browse/KAFKA-1016
> > > >>>>>
> > > >>>>> Guozhang
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Thu, Apr 10, 2014 at 9:21 PM, Arjun <arjun@socialtwist.com>
> > > wrote:
> > > >>>>>
> > > >>>>>   I hope this one would give u  a better idea.
> > > >>>>>
> > > >>>>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
--group
> > > >>>>>> group1
> > > >>>>>> --zkconnect zkhost:port --topic testtopic
> > > >>>>>> Group           Topic                          Pid
Offset
> logSize
> > > >>>>>> Lag             Owner
> > > >>>>>> group1          testtopic    0   253             253
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-0
> > > >>>>>> group1          testtopic    1   267             267
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-0
> > > >>>>>> group1          testtopic    2   254             254
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-0
> > > >>>>>> group1          testtopic    3   265             265
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-0
> > > >>>>>> group1          testtopic    4   261             261
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-1
> > > >>>>>> group1          testtopic    5   294             294
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-1
> > > >>>>>> group1          testtopic    6   248             248
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-1
> > > >>>>>> group1          testtopic    7   271             271
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-1
> > > >>>>>> group1          testtopic    8   240             240
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-2
> > > >>>>>> group1          testtopic    9   261             261
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-2
> > > >>>>>> group1          testtopic    10  290             290
0
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-2
> > > >>>>>> group1          testtopic    11  250             251
1
> > > >>>>>> group1_ip-xx-1397188061429-b5ff1205-2
> > > >>>>>>
> > > >>>>>> If you see the output, in the last line the lag is
1 for that
> > > >>>>>> partition.
> > > >>>>>> I
> > > >>>>>> just send one message. This topic is not new as you
see there
> are
> > > lot
> > > >>>>>> of
> > > >>>>>> messages which have accumlated from yesterday. This
one message
> > will
> > > >>>>>> not
> > > >>>>>> be
> > > >>>>>> consumed by consumer what so ever. But if i send some
10
> messages
> > > then
> > > >>>>>> all
> > > >>>>>> the messages are consumed.
> > > >>>>>>
> > > >>>>>> Please let me know if i have to change any consumer
properties.
> > > >>>>>>
> > > >>>>>> My consumer properties are :
> > > >>>>>> "fetch.wait.max.ms"="180000"
> > > >>>>>> "fetch.min.bytes" = "1"
> > > >>>>>> "auto.offset.reset" = "smallest"
> > > >>>>>> "auto.commit.enable"=  "false"
> > > >>>>>> "fetch.message.max.bytes" = "1048576"
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>> Arjun Narasimha Kota
> > > >>>>>> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
> > > >>>>>>
> > > >>>>>>   The consumer uses do specific topics.
> > > >>>>>>
> > > >>>>>>> On Apr 11, 2014 6:23 AM, "Arjun Kota" <arjun@socialtwist.com
> > > <mailto:
> > > >>>>>>> arjun@socialtwist.com>> wrote:
> > > >>>>>>>
> > > >>>>>>>       Yes the message shows up on the server.
> > > >>>>>>>
> > > >>>>>>>       On Apr 11, 2014 12:07 AM, "Guozhang Wang"
<
> > > wangguoz@gmail.com
> > > >>>>>>>       <mailto:wangguoz@gmail.com>> wrote:
> > > >>>>>>>
> > > >>>>>>>           Hi Arjun,
> > > >>>>>>>
> > > >>>>>>>           If you only send one message, does that
message show
> up
> > > on
> > > >>>>>>> the
> > > >>>>>>>           server? Does
> > > >>>>>>>           you consumer use wildcard topics or
specific topics?
> > > >>>>>>>
> > > >>>>>>>           Guozhang
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>           On Thu, Apr 10, 2014 at 9:20 AM, Arjun
<
> > > >>>>>>> arjun@socialtwist.com
> > > >>>>>>>           <mailto:arjun@socialtwist.com>>
wrote:
> > > >>>>>>>
> > > >>>>>>>           > But  we have auto offset reset
to smallest not
> > largest,
> > > >>>>>>> even
> > > >>>>>>>           then this
> > > >>>>>>>           > issue arises? If so is there any
work around?
> > > >>>>>>>           >
> > > >>>>>>>           > Thanks
> > > >>>>>>>           > Arjun NArasimha Kota
> > > >>>>>>>           >
> > > >>>>>>>           >
> > > >>>>>>>           > On Thursday 10 April 2014 09:39
PM, Guozhang Wang
> > > wrote:
> > > >>>>>>>           >
> > > >>>>>>>           >> It could be https://issues.apache.org/
> > > >>>>>>> jira/browse/KAFKA-1006.
> > > >>>>>>>           >>
> > > >>>>>>>           >> Guozhang
> > > >>>>>>>           >>
> > > >>>>>>>           >>
> > > >>>>>>>           >> On Thu, Apr 10, 2014 at 8:50
AM, Arjun
> > > >>>>>>>           <arjun@socialtwist.com <mailto:arjun@socialtwist.com
> >>
> > > >>>>>>> wrote:
> > > >>>>>>>           >>
> > > >>>>>>>           >>  its auto created
> > > >>>>>>>           >>> but even after topic creation
this is the
> scenario
> > > >>>>>>>           >>>
> > > >>>>>>>           >>> Arjun
> > > >>>>>>>           >>>
> > > >>>>>>>           >>> On Thursday 10 April 2014
08:41 PM, Guozhang Wang
> > > >>>>>>> wrote:
> > > >>>>>>>           >>>
> > > >>>>>>>           >>>  Hi Arjun,
> > > >>>>>>>           >>>>
> > > >>>>>>>           >>>> Did you manually create
the topic or use
> > > >>>>>>> auto.topic.creation?
> > > >>>>>>>           >>>>
> > > >>>>>>>           >>>> Guozhang
> > > >>>>>>>           >>>>
> > > >>>>>>>           >>>>
> > > >>>>>>>           >>>> On Thu, Apr 10, 2014
at 7:39 AM, Arjun
> > > >>>>>>>           <arjun@socialtwist.com <mailto:arjun@socialtwist.com
> >>
> > > >>>>>>> wrote:
> > > >>>>>>>           >>>>
> > > >>>>>>>           >>>>   Hi,
> > > >>>>>>>           >>>>
> > > >>>>>>>           >>>>> We have 3 node
kafka 0.8 setup with zookeepers
> > > >>>>>>> ensemble.
> > > >>>>>>>           We use high
> > > >>>>>>>           >>>>> level
> > > >>>>>>>           >>>>> consumer with auto
commit offset false. I am
> > facing
> > > >>>>>>> some
> > > >>>>>>>           peculiar
> > > >>>>>>>           >>>>> problem
> > > >>>>>>>           >>>>> with kafka. When
i send some 10-20 messages or
> so
> > > the
> > > >>>>>>>           consumer starts
> > > >>>>>>>           >>>>> to
> > > >>>>>>>           >>>>> consume the messages.
But if  i send only one
> > > >>>>>>> message to
> > > >>>>>>>           kafka, then
> > > >>>>>>>           >>>>> even
> > > >>>>>>>           >>>>> though consumer
is active it is not trying to
> > fetch
> > > >>>>>>> the
> > > >>>>>>>           message. There
> > > >>>>>>>           >>>>> is
> > > >>>>>>>           >>>>> nothing in logs,
just the messages are being
> > > fetched
> > > >>>>>>> by
> > > >>>>>>>           the kafka
> > > >>>>>>>           >>>>> consumer.
> > > >>>>>>>           >>>>> The messages are
there in the Kafka server. Can
> > > some
> > > >>>>>>> one
> > > >>>>>>>           let me know
> > > >>>>>>>           >>>>> where
> > > >>>>>>>           >>>>> i am doing wrong.
> > > >>>>>>>           >>>>>
> > > >>>>>>>           >>>>>
> > > >>>>>>>           >>>>> Thanks
> > > >>>>>>>           >>>>> Arjun Narasimha
Kota
> > > >>>>>>>           >>>>>
> > > >>>>>>>           >>>>>
> > > >>>>>>>           >>>>>
> > > >>>>>>>           >>>>
> > > >>>>>>>           >>
> > > >>>>>>>           >
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>           --
> > > >>>>>>>           -- Guozhang
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message