kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Magnus Edenhill <mag...@edenhill.se>
Subject Re: Updated kafka client (producer and consumer)
Date Fri, 06 Dec 2013 10:02:44 GMT
Produce requests in 0.7 are fire-and-forget since there is no
acknowledgements being returned by the broker.
0.8 fixes this by the optional use of required.acks.

I dont know if a non-JVM client is relevant for your needs, but librdkafka
implements all of your listed requirements for 0.8:

1. Message delivery reports (success or specific error) per message.
2.A Prodcuer is fully async, requests are queued locally and sent in
batches to the broker. Multiple requests are usually in-flight at any given
time.
2.B Consumer is fully async. The consumer strives to keep at
least queued.min.messages in the local queue of fetched message ready for
the application to consume.
3. A bit vague, but the application may state what offset to start
consuming from.

See this for more information
https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md


/Magnus



2013/12/6 Tom Brown <tombrown52@gmail.com>

> In our environment we use currently use Kafka 0.7.1.
>
> The core features I am looking for in a client are this:
>
> 1. Provide confirmation of produce requests (or notification of
> disconnection during requests).
> 2. Uses asynchronous IO so that:
>   A. Multiple ops can be queued/in-flight at once.
>   B. Fetch #2 can be in-flight while fetch #1 is being processed (without
> requiring an extra thread per consumer)
> 3. Allow precise control of offsets (like the SimpleConsumer).
>
> I have been working on a version of the Kafka client that meets those
> requirements. However, I recall hearing talk of rewriting the client. Did
> this happen as part of 0.8.x? If so, how different is it from the 0.7.x
> clients, and how well would it support the requirements listed above?
>
> Thanks in advance!
>
> --Tom
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message