kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: New Producer - ONLY sync mode?
Date Mon, 02 Feb 2015 20:13:29 GMT
Yeah as Gwen says there is no sync/async mode anymore. There is a new
configuration which does a lot of what async did in terms of allowing
batching:

batch.size - This is the target amount of data per partition the server
will attempt to batch together.
linger.ms - This is the time the producer will wait for more data to be
sent to better batch up writes. The default is 0 (send immediately). So if
you set this to 50 ms the client will send immediately if it has already
filled up its batch, otherwise it will wait to accumulate the number of
bytes given by batch.size.

To send asynchronously you do
   producer.send(record)
whereas to block on a response you do
   producer.send(record).get();
which will wait for acknowledgement from the server.

One advantage of this model is that the client will do it's best to batch
under the covers even if linger.ms=0. It will do this by batching any data
that arrives while another send is in progress into a single
request--giving a kind of "group commit" effect.

The hope is that this will be both simpler to understand (a single api that
always works the same) and more powerful (you always get a response with
error and offset information whether or not you choose to use it).

-Jay


On Mon, Feb 2, 2015 at 11:15 AM, Gwen Shapira <gshapira@cloudera.com> wrote:

> If you want to emulate the old sync producer behavior, you need to set
> the batch size to 1  (in producer config) and wait on the future you
> get from Send (i.e. future.get)
>
> I can't think of good reasons to do so, though.
>
> Gwen
>
>
> On Mon, Feb 2, 2015 at 11:08 AM, Otis Gospodnetic
> <otis.gospodnetic@gmail.com> wrote:
> > Hi,
> >
> > Is the plan for New Producer to have ONLY async mode?  I'm asking because
> > of this info from the Wiki:
> >
> >
> >    - The producer will always attempt to batch data and will always
> >    immediately return a SendResponse which acts as a Future to allow the
> >    client to await the completion of the request.
> >
> >
> > The word "always" makes me think there will be no sync mode.
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message