kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shahab <shahab.mok...@gmail.com>
Subject Re: How to read in batch using HighLevel Consumer?
Date Wed, 05 Aug 2015 07:03:01 GMT
I just wonder if it is possible to read as batch using SimpleConsumer
instead of HighLevel consumer? does same principle apply to low level
consumer (i.e. SimpleConsumer)?

best,
/Shahab

On Tue, Aug 4, 2015 at 9:10 PM, Gwen Shapira <gwen@confluent.io> wrote:

> To add some internals, the high level consumer actually does read entire
> batches from Kafka. It just exposes them to the user in an event loop,
> because its a very natural API. Users can then batch events the way they
> prefer.
>
> So if you are worried about batches being more efficient than single
> events, you are covered!
>
> Gwen
>
> On Tue, Aug 4, 2015 at 12:04 PM, shahab <shahab.mokari@gmail.com> wrote:
>
> > Thanks a lot Shaminder for clarification and thanks Raja for pointing me
> to
> > the example.
> >
> > best,
> > /shahab
> >
> > On Tue, Aug 4, 2015 at 6:06 PM, Rajasekar Elango <relango@salesforce.com
> >
> > wrote:
> >
> > > Here is an example on what sharninder suggested
> > >
> > >
> >
> http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
> > >
> > > Thanks,
> > > Raja.
> > >
> > > On Tue, Aug 4, 2015 at 12:01 PM, Sharninder <sharninder@gmail.com>
> > wrote:
> > >
> > > > You can't. Kafka is essentially a queue, so you always read messages
> > one
> > > > by one. What you can do is disable auto offset commit, read 100
> > messages,
> > > > process them and then manually commit offset.
> > > >
> > > > --
> > > > Sharninder
> > > >
> > > > > On 04-Aug-2015, at 9:07 pm, shahab <shahab.mokari@gmail.com>
> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > While we the producer can put data as batch in kafka server,  I
> > > couldn't
> > > > > find any API (or any document) saying how we can fetch data as
> batch
> > > from
> > > > > Kafka ?
> > > > > Even when data is placed as batch in kafka server, still using High
> > > Level
> > > > > consumer I can only read one by one, and I can not specify. for
> > > example,
> > > > > read 100 items at once!
> > > > >
> > > > > Is this correct observation? or I am missing something?
> > > > >
> > > > > best,
> > > > > /Shahab
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Raja.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message