kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JIEFU GONG <jg...@berkeley.edu>
Subject Re: How to read in batch using HighLevel Consumer?
Date Wed, 05 Aug 2015 16:16:20 GMT
I believe you can read in batches with the SimpleConsumer by specifying how
many bytes of data you want to process when instantiating a batch request.
Something like:

FetchRequest req = new
FetchRequestBuilder().clientId("some-id").addFetch("topic", partition,
offset, bytesize).build();

On Wed, Aug 5, 2015 at 12:03 AM, shahab <shahab.mokari@gmail.com> wrote:

> I just wonder if it is possible to read as batch using SimpleConsumer
> instead of HighLevel consumer? does same principle apply to low level
> consumer (i.e. SimpleConsumer)?
>
> best,
> /Shahab
>
> On Tue, Aug 4, 2015 at 9:10 PM, Gwen Shapira <gwen@confluent.io> wrote:
>
> > To add some internals, the high level consumer actually does read entire
> > batches from Kafka. It just exposes them to the user in an event loop,
> > because its a very natural API. Users can then batch events the way they
> > prefer.
> >
> > So if you are worried about batches being more efficient than single
> > events, you are covered!
> >
> > Gwen
> >
> > On Tue, Aug 4, 2015 at 12:04 PM, shahab <shahab.mokari@gmail.com> wrote:
> >
> > > Thanks a lot Shaminder for clarification and thanks Raja for pointing
> me
> > to
> > > the example.
> > >
> > > best,
> > > /shahab
> > >
> > > On Tue, Aug 4, 2015 at 6:06 PM, Rajasekar Elango <
> relango@salesforce.com
> > >
> > > wrote:
> > >
> > > > Here is an example on what sharninder suggested
> > > >
> > > >
> > >
> >
> http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
> > > >
> > > > Thanks,
> > > > Raja.
> > > >
> > > > On Tue, Aug 4, 2015 at 12:01 PM, Sharninder <sharninder@gmail.com>
> > > wrote:
> > > >
> > > > > You can't. Kafka is essentially a queue, so you always read
> messages
> > > one
> > > > > by one. What you can do is disable auto offset commit, read 100
> > > messages,
> > > > > process them and then manually commit offset.
> > > > >
> > > > > --
> > > > > Sharninder
> > > > >
> > > > > > On 04-Aug-2015, at 9:07 pm, shahab <shahab.mokari@gmail.com>
> > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > While we the producer can put data as batch in kafka server,
 I
> > > > couldn't
> > > > > > find any API (or any document) saying how we can fetch data
as
> > batch
> > > > from
> > > > > > Kafka ?
> > > > > > Even when data is placed as batch in kafka server, still using
> High
> > > > Level
> > > > > > consumer I can only read one by one, and I can not specify.
for
> > > > example,
> > > > > > read 100 items at once!
> > > > > >
> > > > > > Is this correct observation? or I am missing something?
> > > > > >
> > > > > > best,
> > > > > > /Shahab
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Raja.
> > > >
> > >
> >
>



-- 

Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences

jgong@berkeley.edu <eliselim@berkeley.edu> | (925) 400-3427

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message