kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <g...@confluent.io>
Subject Re: How to read in batch using HighLevel Consumer?
Date Tue, 04 Aug 2015 19:10:34 GMT
To add some internals, the high level consumer actually does read entire
batches from Kafka. It just exposes them to the user in an event loop,
because its a very natural API. Users can then batch events the way they
prefer.

So if you are worried about batches being more efficient than single
events, you are covered!

Gwen

On Tue, Aug 4, 2015 at 12:04 PM, shahab <shahab.mokari@gmail.com> wrote:

> Thanks a lot Shaminder for clarification and thanks Raja for pointing me to
> the example.
>
> best,
> /shahab
>
> On Tue, Aug 4, 2015 at 6:06 PM, Rajasekar Elango <relango@salesforce.com>
> wrote:
>
> > Here is an example on what sharninder suggested
> >
> >
> http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
> >
> > Thanks,
> > Raja.
> >
> > On Tue, Aug 4, 2015 at 12:01 PM, Sharninder <sharninder@gmail.com>
> wrote:
> >
> > > You can't. Kafka is essentially a queue, so you always read messages
> one
> > > by one. What you can do is disable auto offset commit, read 100
> messages,
> > > process them and then manually commit offset.
> > >
> > > --
> > > Sharninder
> > >
> > > > On 04-Aug-2015, at 9:07 pm, shahab <shahab.mokari@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > While we the producer can put data as batch in kafka server,  I
> > couldn't
> > > > find any API (or any document) saying how we can fetch data as batch
> > from
> > > > Kafka ?
> > > > Even when data is placed as batch in kafka server, still using High
> > Level
> > > > consumer I can only read one by one, and I can not specify. for
> > example,
> > > > read 100 items at once!
> > > >
> > > > Is this correct observation? or I am missing something?
> > > >
> > > > best,
> > > > /Shahab
> > >
> >
> >
> >
> > --
> > Thanks,
> > Raja.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message