kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com>
Subject Re: New Producer Public API
Date Wed, 29 Jan 2014 22:41:05 GMT
Hey Guys,

My 2c.

1. RecordSend is a confusing name to me. Shouldn't it be
RecordSendResponse?
2. Random nit: it's annoying to have the Javadoc info for the contstants
on 
http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerConfig.h
tml, but the string constant values on
http://empathybox.com/kafka-javadoc/constant-values.html#kafka.clients.prod
ucer.ProducerConfig.MAX_REQUEST_SIZE_CONFIG. Find myself toggling between
the two a lot. Not sure if this can be fixed easily.
3. MAX_PARTITION_SIZE_CONFIG - this name is kind of confusing.
Specifically, use of the term "partition". thought it was related to Kafka
topic partitions, not grouping together/batching.
4. METADATA_FETCH_TIMEOUT_CONFIG - what happens if this timeout is
exceeded? Do we get an exception on send()?
5. METADATA_REFRESH_MS_CONFIG - why is this useful? Has to do with acks=0,
right? Worth documenting, I think.
6. PARTITIONER_CLASS_CONFIG - link to partitioner interface in javadocs.
Also, missing a period.
7. KafkaProducer.html - send() documentation says "archive" when you mean
achieve, I think.
8. No javadoc for ProduceRequestResult.
9. In ProduceRequestResult, I understand baseOffset to be the first offset
of the set. Is it possible to get the last offset, as well? If I send
messages A, B, C, D, I'm most interested in D's offset.
10. In ProduceRequestResult, prefer Java-bean style (getError,
isCompleted).
11. At first glance, I like option 1A in your serialization list.
12. We should definitely not introduce a ZK dependency for bootstrapping
broker host/ports.
13. No favor on the Future discussion. I really^Int.Max hate checked
exceptions, but I also like standard interfaces. It's a wash in my book.


Cheers,
Chris

On 1/29/14 10:34 AM, "Neha Narkhede" <neha.narkhede@gmail.com> wrote:

>>> The challenge of directly exposing ProduceRequestResult is that the
>offset
>provided is just the base offset and there is no way to know for a
>particular message where it was in relation to that base offset because
>the
>batching is transparent and non-deterministic.
>
>That's a good point. I need to look into the code more closely to see if
>it
>is possible to expose
>something like Future<RequestResult> send(...) where RequestResult has the
>right metadata
>as well as helper APIs that the user would want. For example
>
>Future<RequestResult> messageResponse;
>try {
>  messageResponse = send(...)
>} catch(InterruptedException ie) {
>} catch(ExecutionException ee) {
>}
>
>if(messageResponse.hasError())
>  // handle error
>else {
>   String topic = messageResponse.topic();
>   int partition = messageResponse.partition();
>   long offset = messageResponse.offset();   // can this offset return the
>absolute offset instead of just the relative offset?
>   ...
>}
>
>I could've missed some reasons why we can't do the above. I just think
>that
>separating the future-like functionality of RecordSend
>from the actual response metadata could be useful while supporting Future
>at the same time.
>
>Thanks,
>Neha
>
>
>
>On Wed, Jan 29, 2014 at 10:23 AM, Tom Brown <tombrown52@gmail.com> wrote:
>
>> I strongly support the user of Future. In fact, the cancel method may
>>not
>> be useless. Since the producer is meant to be used by N threads, it
>>could
>> easily get overloaded such that a produce request could not be sent
>> immediately and had to be queued. In that case, cancelling should cause
>>it
>> to not actually get sent.
>>
>> --Tom
>>
>>
>> On Wed, Jan 29, 2014 at 11:06 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>>
>> > Hey Neha,
>> >
>> > Error handling in RecordSend works as in Future you will get the
>> exception
>> > if there is one from any of the accessor methods or await().
>> >
>> > The purpose of hasError was that you can write things slightly more
>> simply
>> > (which some people expressed preference for):
>> >   if(send.hasError())
>> >     // do something
>> >   long offset = send.offset();
>> >
>> > Instead of the more the slightly longer:
>> > try {
>> >    long offset = send.offset();
>> > } catch (KafkaException e) {
>> >    // do something
>> > }
>> >
>> >
>> > On Wed, Jan 29, 2014 at 10:01 AM, Neha Narkhede
>><neha.narkhede@gmail.com
>> > >wrote:
>> >
>> > > Regarding the use of Futures -
>> > >
>> > > Agree that there are some downsides to using Futures but both
>> approaches
>> > > have some tradeoffs.
>> > >
>> > > - Standardization and usability
>> > > Future is a widely used and understood Java API and given that the
>> > > functionality that RecordSend hopes to provide is essentially that
>>of
>> > > Future, I think it makes sense to expose a widely understood public
>>API
>> > for
>> > > our clients. RecordSend, on the other hand, seems to provide some
>>APIs
>> > that
>> > > are very similar to that of Future, in addition to exposing a bunch
>>of
>> > APIs
>> > > that belong to ProduceRequestResult. As a user, I would've really
>> > preferred
>> > > to deal with ProduceRequestResult directly -
>> > > Future<ProduceRequestResult> send(...)
>> > >
>> > > - Error handling
>> > > RecordSend's error handling is quite unintuitive where the user has
>>to
>> > > remember to invoke hasError and error, instead of just throwing the
>> > > exception. Now there are
>> > > some downsides regarding error handling with the Future as well,
>>where
>> > the
>> > > user has to catch InterruptedException when we would never run into
>>it.
>> > > However, it seems like a price worth paying for supporting a
>>standard
>> API
>> > > and error handling
>> > >
>> > > - Unused APIs
>> > > This is a downside of using Future, where the cancel() operation
>>would
>> > > always return false and mean nothing. But we can mention that
>>caveat in
>> > our
>> > > Java docs.
>> > >
>> > > To summarize, I would prefer to expose a well understood and widely
>> > adopted
>> > > Java API and put up with the overhead of catching one unnecessary
>> checked
>> > > exception, rather than wrap the useful ProduceRequestResult in a
>>custom
>> > > async object (RecordSend) and explain that to our many users.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Jan 28, 2014 at 8:10 PM, Jay Kreps <jay.kreps@gmail.com>
>> wrote:
>> > >
>> > > > Hey Neha,
>> > > >
>> > > > Can you elaborate on why you prefer using Java's Future? The
>>downside
>> > in
>> > > my
>> > > > mind is the use of the checked InterruptedException and
>> > > ExecutionException.
>> > > > ExecutionException is arguable, but forcing you to catch
>> > > > InterruptedException, often in code that can't be interrupted,
>>seems
>> > > > perverse. It also leaves us with the cancel() method which I don't
>> > think
>> > > we
>> > > > really can implement.
>> > > >
>> > > > Option 1A, to recap/elaborate, was the following. There is no
>> > Serializer
>> > > or
>> > > > Partitioner api. We take a byte[] key and value and an optional
>> integer
>> > > > partition. If you specify the integer partition it will be used.
>>If
>> you
>> > > do
>> > > > not specify a key or a partition the partition will be chosen in a
>> > round
>> > > > robin fashion. If you specify a key but no partition we will
>>chose a
>> > > > partition based on a hash of the key. In order to let the user
>>find
>> the
>> > > > partition we will need to given them access to the Cluster
>>instance
>> > > > directly from the producer.
>> > > >
>> > > > -Jay
>> > > >
>> > > >
>> > > > On Tue, Jan 28, 2014 at 6:25 PM, Neha Narkhede <
>> > neha.narkhede@gmail.com
>> > > > >wrote:
>> > > >
>> > > > > Here are more thoughts on the public APIs -
>> > > > >
>> > > > > - I suggest we use java's Future instead of custom Future
>> especially
>> > > > since
>> > > > > it is part of the public API
>> > > > >
>> > > > > - Serialization: I like the simplicity of the producer APIs with
>> the
>> > > > > absence of serialization where we just deal with byte arrays for
>> keys
>> > > and
>> > > > > values. What I don't like about this is the performance
>>overhead on
>> > the
>> > > > > Partitioner for any kind of custom partitioning based on the
>> > > > partitionKey.
>> > > > > Since the only purpose of partitionKey is to do custom
>> partitioning,
>> > > why
>> > > > > can't we take it in directly as an integer and let the user
>>figure
>> > out
>> > > > the
>> > > > > mapping from partition_key -> partition_id using the
>>getCluster()
>> > API?
>> > > > If I
>> > > > > understand correctly, this is similar to what you suggested as
>>part
>> > of
>> > > > > option 1A. I like this approach since it maintains the
>>simplicity
>> of
>> > > APIs
>> > > > > by allowing us to deal with bytes and does not compromise
>> performance
>> > > in
>> > > > > the custom partitioning case.
>> > > > >
>> > > > > Thanks,
>> > > > > Neha
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, Jan 28, 2014 at 5:42 PM, Jay Kreps <jay.kreps@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Hey Tom,
>> > > > > >
>> > > > > > That sounds cool. How did you end up handling parallel I/O if
>>you
>> > > wrap
>> > > > > the
>> > > > > > individual connections? Don't you need some selector that
>>selects
>> > > over
>> > > > > all
>> > > > > > the connections?
>> > > > > >
>> > > > > > -Jay
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Jan 28, 2014 at 2:31 PM, Tom Brown
>><tombrown52@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > > > I implemented a 0.7 client in pure java, and its API very
>> closely
>> > > > > > resembled
>> > > > > > > this. (When multiple people independently engineer the same
>> > > solution,
>> > > > > > it's
>> > > > > > > probably good... right?). However, there were a few
>> architectural
>> > > > > > > differences with my client:
>> > > > > > >
>> > > > > > > 1. The basic client itself was just an asynchronous layer
>> around
>> > > the
>> > > > > > > different server functions. In and of itself it had no
>> knowledge
>> > of
>> > > > > > > partitions, only servers (and maintained TCP connections to
>> > them).
>> > > > > > >
>> > > > > > > 2. The main producer was an additional layer that provided a
>> > > > high-level
>> > > > > > > interface that could batch individual messages based on
>> > partition.
>> > > > > > >
>> > > > > > > 3. Knowledge of partitioning was done via an interface so
>>that
>> > > > > different
>> > > > > > > strategies could be used.
>> > > > > > >
>> > > > > > > 4. Partitioning was done by the user, with knowledge of the
>> > > available
>> > > > > > > partitions provided by #3.
>> > > > > > >
>> > > > > > > 5. Serialization was done by the user to simplify the API.
>> > > > > > >
>> > > > > > > 6. Futures were used to make asynchronous emulate
>>synchronous
>> > > calls.
>> > > > > > >
>> > > > > > >
>> > > > > > > The main benefit of this approach is flexibility. For
>>example,
>> > > since
>> > > > > the
>> > > > > > > base client was just a managed connection (and not
>>inherently a
>> > > > > > producer),
>> > > > > > > it was easy to composite a produce request and an offsets
>> request
>> > > > > > together
>> > > > > > > into a confirmed produce request (officially not available
>>in
>> > 0.7).
>> > > > > > >
>> > > > > > > Decoupling the basic client from partition management
>>allowed
>> the
>> > > me
>> > > > to
>> > > > > > > implement zk discovery as a separate project so that the
>>main
>> > > project
>> > > > > had
>> > > > > > > no complex dependencies. The same was true of decoupling
>> > > > serialization.
>> > > > > > > It's trivial to build an optional layer that adds those
>> features
>> > > in,
>> > > > > > while
>> > > > > > > allowing access to the base APIs for those that need it.
>> > > > > > >
>> > > > > > > Using standard Future objects was also beneficial, since I
>> could
>> > > > > combine
>> > > > > > > them with existing tools (such as guava).
>> > > > > > >
>> > > > > > > It may be too late to be of use, but I have been working
>>with
>> my
>> > > > > > company's
>> > > > > > > legal department to release the implementation I described
>> above.
>> > > If
>> > > > > > you're
>> > > > > > > interested in it, let me know.
>> > > > > > >
>> > > > > > >
>> > > > > > > To sum up my thoughts regarding the new API, I think it's a
>> great
>> > > > > start.
>> > > > > > I
>> > > > > > > would like to see a more layered approach so I can use the
>> parts
>> > I
>> > > > > want,
>> > > > > > > and adapt the other parts as needed. I would also like to
>>see
>> > > > standard
>> > > > > > > interfaces (especially Future) used where they makes sense.
>> > > > > > >
>> > > > > > > --Tom
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Tue, Jan 28, 2014 at 1:33 PM, Roger Hoover <
>> > > > roger.hoover@gmail.com
>> > > > > > > >wrote:
>> > > > > > >
>> > > > > > > > +1 ListenableFuture: If this works similar to Deferreds in
>> > > Twisted
>> > > > > > Python
>> > > > > > > > or Promised IO in Javascript, I think this is a great
>>pattern
>> > for
>> > > > > > > > decoupling your callback logic from the place where the
>> Future
>> > is
>> > > > > > > > generated.  You can register as many callbacks as you
>>like,
>> > each
>> > > in
>> > > > > the
>> > > > > > > > appropriate layer of the code and have each observer get
>> > notified
>> > > > > when
>> > > > > > > the
>> > > > > > > > promised i/o is complete without any of them knowing about
>> each
>> > > > > other.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, Jan 28, 2014 at 11:32 AM, Jay Kreps <
>> > jay.kreps@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hey Ross,
>> > > > > > > > >
>> > > > > > > > > - ListenableFuture: Interesting. That would be an
>> alternative
>> > > to
>> > > > > the
>> > > > > > > > direct
>> > > > > > > > > callback support we provide. There could be pros to
>>this,
>> let
>> > > me
>> > > > > > think
>> > > > > > > > > about it.
>> > > > > > > > > - We could provide layering, but I feel that the
>> > serialization
>> > > is
>> > > > > > such
>> > > > > > > a
>> > > > > > > > > small thing we should just make a decision and chose
>>one,
>> it
>> > > > > doesn't
>> > > > > > > seem
>> > > > > > > > > to me to justify a whole public facing layer.
>> > > > > > > > > - Yes, this is fairly esoteric, essentially I think it
>>is
>> > > fairly
>> > > > > > > similar
>> > > > > > > > to
>> > > > > > > > > databases like DynamoDB that allow you to specify two
>> > partition
>> > > > > keys
>> > > > > > (I
>> > > > > > > > > think DynamoDB does this...). The reasoning is that in
>>fact
>> > > there
>> > > > > are
>> > > > > > > > > several things you can use the key field for: (1) to
>> compute
>> > > the
>> > > > > > > > partition
>> > > > > > > > > to store the data in, (2) as a unique identifier to
>> > deduplicate
>> > > > > that
>> > > > > > > > > partition's records within a log. These two things are
>> almost
>> > > > > always
>> > > > > > > the
>> > > > > > > > > same, but occationally may differ when you want to group
>> data
>> > > in
>> > > > a
>> > > > > > more
>> > > > > > > > > sophisticated way then just a hash of the primary key
>>but
>> > still
>> > > > > > retain
>> > > > > > > > the
>> > > > > > > > > proper primary key for delivery to the consumer and log
>> > > > compaction.
>> > > > > > > > >
>> > > > > > > > > -Jay
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Tue, Jan 28, 2014 at 3:24 AM, Ross Black <
>> > > > > ross.w.black@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Jay,
>> > > > > > > > > >
>> > > > > > > > > > - Just to add some more info/confusion about possibly
>> using
>> > > > > Future
>> > > > > > > ...
>> > > > > > > > > >   If Kafka uses a JDK future, it plays nicely with
>>other
>> > > > > frameworks
>> > > > > > > as
>> > > > > > > > > > well.
>> > > > > > > > > >   Google Guava has a ListenableFuture that allows
>> callback
>> > > > > handling
>> > > > > > > to
>> > > > > > > > be
>> > > > > > > > > > added via the returned future, and allows the
>>callbacks
>> to
>> > be
>> > > > > > passed
>> > > > > > > > off
>> > > > > > > > > to
>> > > > > > > > > > a specified executor.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> 
>>http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/co
>>m/google/common/util/concurrent/ListenableFuture.html
>> > > > > > > > > >   The JDK future can easily be converted to a
>>listenable
>> > > > future.
>> > > > > > > > > >
>> > > > > > > > > > - On the question of byte[] vs Object, could this be
>> solved
>> > > by
>> > > > > > > layering
>> > > > > > > > > the
>> > > > > > > > > > API?  eg. a raw producer (use byte[] and specify the
>> > > partition
>> > > > > > > number)
>> > > > > > > > > and
>> > > > > > > > > > a normal producer (use generic object and specify a
>> > > > Partitioner)?
>> > > > > > > > > >
>> > > > > > > > > > - I am confused by the keys in ProducerRecord and
>> > > Partitioner.
>> > > > > >  What
>> > > > > > > is
>> > > > > > > > > the
>> > > > > > > > > > usage for both a key and a partition key? (I am not
>>yet
>> > using
>> > > > > 0.8)
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Ross
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 28 January 2014 05:00, Xavier Stevens <
>> > xavier@gaikai.com>
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > AutoCloseable would be nice for us as most of our
>>code
>> is
>> > > > using
>> > > > > > > Java
>> > > > > > > > 7
>> > > > > > > > > at
>> > > > > > > > > > > this point.
>> > > > > > > > > > >
>> > > > > > > > > > > I like Dropwizard's configuration mapping to POJOs
>>via
>> > > > Jackson,
>> > > > > > but
>> > > > > > > > if
>> > > > > > > > > > you
>> > > > > > > > > > > wanted to stick with property maps I don't care
>>enough
>> to
>> > > > > object.
>> > > > > > > > > > >
>> > > > > > > > > > > If the producer only dealt with bytes, is there a
>>way
>> we
>> > > > could
>> > > > > > > still
>> > > > > > > > > due
>> > > > > > > > > > > partition plugins without specifying the number
>> > > explicitly? I
>> > > > > > would
>> > > > > > > > > > prefer
>> > > > > > > > > > > to be able to pass in field(s) that would be used by
>> the
>> > > > > > > partitioner.
>> > > > > > > > > > > Obviously if this wasn't possible you could always
>> > > > deserialize
>> > > > > > the
>> > > > > > > > > object
>> > > > > > > > > > > in the partitioner and grab the fields you want, but
>> that
>> > > > seems
>> > > > > > > > really
>> > > > > > > > > > > expensive to do on every message.
>> > > > > > > > > > >
>> > > > > > > > > > > It would also be nice to have a Java API Encoder
>> > > constructor
>> > > > > > taking
>> > > > > > > > in
>> > > > > > > > > > > VerifiableProperties. Scala understands how to
>>handle
>> > > "props:
>> > > > > > > > > > > VerifiableProperties = null", but Java doesn't. So
>>you
>> > > don't
>> > > > > run
>> > > > > > > into
>> > > > > > > > > > this
>> > > > > > > > > > > problem until runtime.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > -Xavier
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Mon, Jan 27, 2014 at 9:37 AM, Clark Breyman <
>> > > > > > clark@breyman.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Jay -
>> > > > > > > > > > > >
>> > > > > > > > > > > > Config - your explanation makes sense. I'm just so
>> > > > accustomed
>> > > > > > to
>> > > > > > > > > having
>> > > > > > > > > > > > Jackson automatically map my configuration
>>objects to
>> > > POJOs
>> > > > > > that
>> > > > > > > > I've
>> > > > > > > > > > > > stopped using property files. They are lingua
>>franca.
>> > The
>> > > > > only
>> > > > > > > > > thought
>> > > > > > > > > > > > might be to separate the config interface from the
>> > > > > > implementation
>> > > > > > > > to
>> > > > > > > > > > > allow
>> > > > > > > > > > > > for alternatives, but that might undermine your
>>point
>> > of
>> > > > "do
>> > > > > it
>> > > > > > > > this
>> > > > > > > > > > way
>> > > > > > > > > > > so
>> > > > > > > > > > > > that everyone can find it where they expect it".
>> > > > > > > > > > > >
>> > > > > > > > > > > > Serialization: Of the options, I like 1A the best,
>> > though
>> > > > > > > possibly
>> > > > > > > > > with
>> > > > > > > > > > > > either an option to specify a partition key rather
>> than
>> > > ID
>> > > > > or a
>> > > > > > > > > helper
>> > > > > > > > > > to
>> > > > > > > > > > > > translate an arbitrary byte[] or long into a
>> partition
>> > > > > number.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks
>> > > > > > > > > > > > Clark
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Sun, Jan 26, 2014 at 9:13 PM, Jay Kreps <
>> > > > > > jay.kreps@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for the detailed thoughts. Let me
>>elaborate
>> on
>> > > the
>> > > > > > > config
>> > > > > > > > > > thing.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I agree that at first glance key-value strings
>> don't
>> > > seem
>> > > > > > like
>> > > > > > > a
>> > > > > > > > > very
>> > > > > > > > > > > > good
>> > > > > > > > > > > > > configuration api for a client. Surely a
>>well-typed
>> > > > config
>> > > > > > > class
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > > > better! I actually disagree and let me see if I
>>can
>> > > > > convince
>> > > > > > > you.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > My reasoning has nothing to do with the api and
>> > > > everything
>> > > > > to
>> > > > > > > do
>> > > > > > > > > with
>> > > > > > > > > > > > > operations.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Clients are embedded in applications which are
>> > > themselves
>> > > > > > > > > configured.
>> > > > > > > > > > > In
>> > > > > > > > > > > > > any place that takes operations seriously the
>> > > > configuration
>> > > > > > for
>> > > > > > > > > these
>> > > > > > > > > > > > > applications will be version controlled and
>> > maintained
>> > > > > > through
>> > > > > > > > some
>> > > > > > > > > > > kind
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > config management system. If we give a config
>>class
>> > > with
>> > > > > > > getters
>> > > > > > > > > and
>> > > > > > > > > > > > > setters the application has to expose those
>> > properties
>> > > to
>> > > > > its
>> > > > > > > > > > > > > configuration. What invariably happens is that
>>the
>> > > > > > application
>> > > > > > > > > > exposes
>> > > > > > > > > > > > only
>> > > > > > > > > > > > > a choice few properties that they thought they
>> would
>> > > > > change.
>> > > > > > > > > > > Furthermore
>> > > > > > > > > > > > > the application will make up a name for these
>> configs
>> > > > that
>> > > > > > > seems
>> > > > > > > > > > > > intuitive
>> > > > > > > > > > > > > at the time in the 2 seconds the engineer spends
>> > > thinking
>> > > > > > about
>> > > > > > > > it.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Now consider the result of this in the large.
>>You
>> end
>> > > up
>> > > > > with
>> > > > > > > > > dozens
>> > > > > > > > > > or
>> > > > > > > > > > > > > hundreds of applications that have the client
>> > embedded.
>> > > > > Each
>> > > > > > > > > exposes
>> > > > > > > > > > a
>> > > > > > > > > > > > > different, inadequate subset of the possible
>> configs,
>> > > > each
>> > > > > > with
>> > > > > > > > > > > different
>> > > > > > > > > > > > > names. It is a nightmare.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > If you use a string-string map the config system
>> can
>> > > > > directly
>> > > > > > > > get a
>> > > > > > > > > > > > bundle
>> > > > > > > > > > > > > of config key-value pairs and put them into the
>> > client.
>> > > > > This
>> > > > > > > > means
>> > > > > > > > > > that
>> > > > > > > > > > > > all
>> > > > > > > > > > > > > configuration is automatically available with
>>the
>> > name
>> > > > > > > documented
>> > > > > > > > > on
>> > > > > > > > > > > the
>> > > > > > > > > > > > > website in every application that does this. If
>>you
>> > > > upgrade
>> > > > > > to
>> > > > > > > a
>> > > > > > > > > new
>> > > > > > > > > > > > kafka
>> > > > > > > > > > > > > version with more configs those will be exposed
>> too.
>> > If
>> > > > you
>> > > > > > > > realize
>> > > > > > > > > > > that
>> > > > > > > > > > > > > you need to change a default you can just go
>> through
>> > > your
>> > > > > > > configs
>> > > > > > > > > and
>> > > > > > > > > > > > > change it everywhere as it will have the same
>>name
>> > > > > > everywhere.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > -Jay
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Sun, Jan 26, 2014 at 4:47 PM, Clark Breyman <
>> > > > > > > > clark@breyman.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thanks Jay. I'll see if I can put together a
>>more
>> > > > > complete
>> > > > > > > > > > response,
>> > > > > > > > > > > > > > perhaps as separate threads so that topics
>>don't
>> > get
>> > > > > > > entangled.
>> > > > > > > > > In
>> > > > > > > > > > > the
>> > > > > > > > > > > > > mean
>> > > > > > > > > > > > > > time, here's a couple responses:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Serialization: you've broken out a sub-thread
>>so
>> > i'll
>> > > > > reply
>> > > > > > > > > there.
>> > > > > > > > > > My
>> > > > > > > > > > > > > bias
>> > > > > > > > > > > > > > is that I like generics (except for
>>type-erasure)
>> > and
>> > > > in
>> > > > > > > > > particular
>> > > > > > > > > > > > they
>> > > > > > > > > > > > > > make it easy to compose serializers for
>>compound
>> > > > payloads
>> > > > > > > (e.g.
>> > > > > > > > > > when
>> > > > > > > > > > > a
>> > > > > > > > > > > > > > common header wraps a payload of parameterized
>> > type).
>> > > > > I'll
>> > > > > > > > > respond
>> > > > > > > > > > to
>> > > > > > > > > > > > > your
>> > > > > > > > > > > > > > 4-options message with an example.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Build: I've seen a lot of "maven-compatible"
>> build
>> > > > > systems
>> > > > > > > > > produce
>> > > > > > > > > > > > > > "artifacts" that aren't really artifacts - no
>> > > embedded
>> > > > > POM
>> > > > > > > or,
>> > > > > > > > > > worst,
>> > > > > > > > > > > > > > malformed POM. I know the sbt-generated
>>artifacts
>> > > were
>> > > > > this
>> > > > > > > > way -
>> > > > > > > > > > > onus
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > on me to see what gradle is spitting out and
>> what a
>> > > > maven
>> > > > > > > build
>> > > > > > > > > > might
>> > > > > > > > > > > > > look
>> > > > > > > > > > > > > > like. Maven may be old and boring, but it gets
>> out
>> > of
>> > > > the
>> > > > > > way
>> > > > > > > > and
>> > > > > > > > > > > > > > integrates really seamlessly with a lot of
>>IDEs.
>> > When
>> > > > > some
>> > > > > > > > scala
>> > > > > > > > > > > > > projects I
>> > > > > > > > > > > > > > was working on in the fall of 2011 switched
>>from
>> > sbt
>> > > to
>> > > > > > > maven,
>> > > > > > > > > > build
>> > > > > > > > > > > > > became
>> > > > > > > > > > > > > > a non-issue.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Config: Not a big deal  and no, I don't think
>>a
>> > > > > dropwizard
>> > > > > > > > > > dependency
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > appropriate. I do like using simple entity
>>beans
>> > > > (POJO's
>> > > > > > not
>> > > > > > > > > j2EE)
>> > > > > > > > > > > for
>> > > > > > > > > > > > > > configuration, especially if they can be
>> marshalled
>> > > > > without
>> > > > > > > > > > > annotation
>> > > > > > > > > > > > by
>> > > > > > > > > > > > > > Jackson. I only mentioned the
>>dropwizard-extras
>> > > >  because
>> > > > > it
>> > > > > > > has
>> > > > > > > > > > some
>> > > > > > > > > > > > > entity
>> > > > > > > > > > > > > > bean versions of the ZK and Kafka configs.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Domain-packaging: Also not a big deal - it's
>> what's
>> > > > > > expected
>> > > > > > > > and
>> > > > > > > > > > it's
>> > > > > > > > > > > > > > pretty free in most IDE's. The advantages I
>>see
>> is
>> > > that
>> > > > > it
>> > > > > > is
>> > > > > > > > > clear
>> > > > > > > > > > > > > whether
>> > > > > > > > > > > > > > something is from the Apache Kafka project and
>> > > whether
>> > > > > > > > something
>> > > > > > > > > is
>> > > > > > > > > > > > from
>> > > > > > > > > > > > > > another org and related to Kafka. That said,
>> > nothing
>> > > > > really
>> > > > > > > > > > enforces
>> > > > > > > > > > > > it.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Futures: I'll see if I can create some
>>examples
>> to
>> > > > > > > demonstrate
>> > > > > > > > > > Future
>> > > > > > > > > > > > > > making interop easier.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Regards,
>> > > > > > > > > > > > > > C
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Fri, Jan 24, 2014 at 4:36 PM, Jay Kreps <
>> > > > > > > > jay.kreps@gmail.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hey Clark,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > - Serialization: Yes I agree with these
>>though
>> I
>> > > > don't
>> > > > > > > > consider
>> > > > > > > > > > the
>> > > > > > > > > > > > > loss
>> > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > generics a big issue. I'll try to summarize
>> what
>> > I
>> > > > > would
>> > > > > > > > > consider
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > best
>> > > > > > > > > > > > > > > alternative api with raw byte[].
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > - Maven: We had this debate a few months
>>back
>> and
>> > > the
>> > > > > > > > consensus
>> > > > > > > > > > was
>> > > > > > > > > > > > > > gradle.
>> > > > > > > > > > > > > > > Is there a specific issue with the poms
>>gradle
>> > > > makes? I
>> > > > > > am
>> > > > > > > > > > > extremely
>> > > > > > > > > > > > > > loath
>> > > > > > > > > > > > > > > to revisit the issue as build issues are a
>> > > recurring
>> > > > > > thing
>> > > > > > > > and
>> > > > > > > > > no
>> > > > > > > > > > > one
>> > > > > > > > > > > > > > ever
>> > > > > > > > > > > > > > > agrees and ultimately our build needs are
>>very
>> > > > simple.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > - Config: I'm not sure if I follow the
>>point.
>> Are
>> > > you
>> > > > > > > saying
>> > > > > > > > we
>> > > > > > > > > > > > should
>> > > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > something in dropwizard for config? One
>> principle
>> > > > here
>> > > > > is
>> > > > > > > to
>> > > > > > > > > try
>> > > > > > > > > > to
>> > > > > > > > > > > > > > remove
>> > > > > > > > > > > > > > > as many client dependencies as possible as
>>we
>> > > > > inevitably
>> > > > > > > run
>> > > > > > > > > into
>> > > > > > > > > > > > > > terrible
>> > > > > > > > > > > > > > > compatibility issues with users who use the
>> same
>> > > > > library
>> > > > > > or
>> > > > > > > > its
>> > > > > > > > > > > > > > > dependencies at different versions. Or are
>>you
>> > > > talking
>> > > > > > > about
>> > > > > > > > > > > > > maintaining
>> > > > > > > > > > > > > > > compatibility with existing config
>>parameters?
>> I
>> > > > think
>> > > > > as
>> > > > > > > > much
>> > > > > > > > > > as a
>> > > > > > > > > > > > > > config
>> > > > > > > > > > > > > > > in the existing client makes sense it should
>> have
>> > > the
>> > > > > > same
>> > > > > > > > name
>> > > > > > > > > > (I
>> > > > > > > > > > > > was
>> > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > bit sloppy about that so I'll fix any errors
>> > > there).
>> > > > > > There
>> > > > > > > > are
>> > > > > > > > > a
>> > > > > > > > > > > few
>> > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > things and we should give those reasonable
>> > > defaults.
>> > > > I
>> > > > > > > think
>> > > > > > > > > > config
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > important so I'll start a thread on the 
>>config
>> > > > package
>> > > > > in
>> > > > > > > > > there.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > - org.apache.kafka: We could do this. I 
>>always
>> > > > > considered
>> > > > > > > it
>> > > > > > > > > kind
>> > > > > > > > > > > of
>> > > > > > > > > > > > an
>> > > > > > > > > > > > > > odd
>> > > > > > > > > > > > > > > thing Java programmers do that has no real
>> > > motivation
>> > > > > > (but
>> > > > > > > I
>> > > > > > > > > > could
>> > > > > > > > > > > be
>> > > > > > > > > > > > > > > re-educated!). I don't think it ends up
>> reducing
>> > > > naming
>> > > > > > > > > conflicts
>> > > > > > > > > > > in
>> > > > > > > > > > > > > > > practice and it adds a lot of noise and 
>>nested
>> > > > > > directories.
>> > > > > > > > Is
>> > > > > > > > > > > there
>> > > > > > > > > > > > a
>> > > > > > > > > > > > > > > reason you prefer this or just to be more
>> > standard?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > - Future: Basically I didn't see any 
>>particular
>> > > > > > advantage.
>> > > > > > > > The
>> > > > > > > > > > > > cancel()
>> > > > > > > > > > > > > > > method doesn't really make sense so probably
>> > > wouldn't
>> > > > > > work.
>> > > > > > > > > > > Likewise
>> > > > > > > > > > > > I
>> > > > > > > > > > > > > > > dislike the checked exceptions it requires.
>> > > > Basically I
>> > > > > > > just
>> > > > > > > > > > wrote
>> > > > > > > > > > > > out
>> > > > > > > > > > > > > > some
>> > > > > > > > > > > > > > > code examples and it seemed cleaner with a
>> > special
>> > > > > > purpose
>> > > > > > > > > > object.
>> > > > > > > > > > > I
>> > > > > > > > > > > > > > wasn't
>> > > > > > > > > > > > > > > actually aware of plans for improved 
>>futures in
>> > > java
>> > > > 8
>> > > > > or
>> > > > > > > the
>> > > > > > > > > > other
>> > > > > > > > > > > > > > > integrations. Maybe you could elaborate on
>> this a
>> > > bit
>> > > > > and
>> > > > > > > > show
>> > > > > > > > > > how
>> > > > > > > > > > > it
>> > > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > be used? Sounds promising, I just don't 
>>know a
>> > lot
>> > > > > about
>> > > > > > > it.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > -Jay
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Fri, Jan 24, 2014 at 3:30 PM, Clark 
>>Breyman
>> <
>> > > > > > > > > > clark@breyman.com>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Jay - Thanks for the call for comments.
>> Here's
>> > > some
>> > > > > > > initial
>> > > > > > > > > > > input:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > - Make message serialization a client
>> > > > responsibility
>> > > > > > > > (making
>> > > > > > > > > > all
>> > > > > > > > > > > > > > messages
>> > > > > > > > > > > > > > > > byte[]). Reflection-based loading makes it
>> > harder
>> > > > to
>> > > > > > use
>> > > > > > > > > > generic
>> > > > > > > > > > > > > codecs
>> > > > > > > > > > > > > > > > (e.g.  Envelope<PREFIX, DATA, SUFFIX>) or
>> build
>> > > up
>> > > > > > codec
>> > > > > > > > > > > > > > > programmatically.
>> > > > > > > > > > > > > > > > Non-default partitioning should require an
>> > > explicit
>> > > > > > > > partition
>> > > > > > > > > > > key.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > - I really like the fact that it will be
>> native
>> > > > Java.
>> > > > > > > > Please
>> > > > > > > > > > > > consider
>> > > > > > > > > > > > > > > using
>> > > > > > > > > > > > > > > > native maven and not sbt, gradle, ivy, 
>>etc as
>> > > they
>> > > > > > don't
>> > > > > > > > > > reliably
>> > > > > > > > > > > > > play
>> > > > > > > > > > > > > > > nice
>> > > > > > > > > > > > > > > > in the maven ecosystem. A jar without a
>> > > well-formed
>> > > > > pom
>> > > > > > > > > doesn't
>> > > > > > > > > > > > feel
>> > > > > > > > > > > > > > > like a
>> > > > > > > > > > > > > > > > real artifact. The pom's generated by sbt 
>>et
>> > al.
>> > > > are
>> > > > > > not
>> > > > > > > > well
>> > > > > > > > > > > > formed.
>> > > > > > > > > > > > > > > Using
>> > > > > > > > > > > > > > > > maven will make builds and IDE integration
>> much
>> > > > > > smoother.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > - Look at Nick Telford's dropwizard-extras
>> > > package
>> > > > in
>> > > > > > > which
>> > > > > > > > > he
>> > > > > > > > > > > > > defines
>> > > > > > > > > > > > > > > some
>> > > > > > > > > > > > > > > > Jackson-compatible POJO's for loading
>> > > > configuration.
>> > > > > > > Seems
>> > > > > > > > > like
>> > > > > > > > > > > > your
>> > > > > > > > > > > > > > > client
>> > > > > > > > > > > > > > > > migration is similar. The config objects
>> should
>> > > > have
>> > > > > > > > > > constructors
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > factories that accept Map<String, String> 
>>and
>> > > > > > Properties
>> > > > > > > > for
>> > > > > > > > > > ease
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > migration.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > - Would you consider using the
>> org.apache.kafka
>> > > > > package
>> > > > > > > for
>> > > > > > > > > the
>> > > > > > > > > > > new
>> > > > > > > > > > > > > API
>> > > > > > > > > > > > > > > > (quibble)
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > - Why create your own futures rather than 
>>use
>> > > > > > > > > > > > > > > > java.util.concurrent.Future<Long> or 
>>similar?
>> > > > > Standard
>> > > > > > > > > futures
>> > > > > > > > > > > will
>> > > > > > > > > > > > > > play
>> > > > > > > > > > > > > > > > nice with other reactive libs and things 
>>like
>> > > J8's
>> > > > > > > > > > > > ComposableFuture.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thanks again,
>> > > > > > > > > > > > > > > > C
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Fri, Jan 24, 2014 at 2:46 PM, Roger
>> Hoover <
>> > > > > > > > > > > > > roger.hoover@gmail.com
>> > > > > > > > > > > > > > > > >wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > A couple comments:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 1) Why does the config use a broker list
>> > > instead
>> > > > of
>> > > > > > > > > > discovering
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > brokers
>> > > > > > > > > > > > > > > > > in ZooKeeper?  It doesn't match the
>> > > > > HighLevelConsumer
>> > > > > > > > API.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 2) It looks like broker connections are
>> > created
>> > > > on
>> > > > > > > > demand.
>> > > > > > > > > >  I'm
>> > > > > > > > > > > > > > > wondering
>> > > > > > > > > > > > > > > > > if sometimes you might want to flush out
>> > config
>> > > > or
>> > > > > > > > network
>> > > > > > > > > > > > > > connectivity
>> > > > > > > > > > > > > > > > > issues before pushing the first message
>> > > through.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Should there also be a
>> > KafkaProducer.connect()
>> > > or
>> > > > > > > .open()
>> > > > > > > > > > > method
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > connectAll()?  I guess it would try to
>> > connect
>> > > to
>> > > > > all
>> > > > > > > > > brokers
>> > > > > > > > > > > in
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > BROKER_LIST_CONFIG
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > HTH,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Roger
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Fri, Jan 24, 2014 at 11:54 AM, Jay
>> Kreps <
>> > > > > > > > > > > jay.kreps@gmail.com
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > As mentioned in a previous email we 
>>are
>> > > working
>> > > > > on
>> > > > > > a
>> > > > > > > > > > > > > > > re-implementation
>> > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > the producer. I would like to use this
>> > email
>> > > > > thread
>> > > > > > > to
>> > > > > > > > > > > discuss
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > details
>> > > > > > > > > > > > > > > > > > of the public API and the 
>>configuration.
>> I
>> > > > would
>> > > > > > love
>> > > > > > > > for
>> > > > > > > > > > us
>> > > > > > > > > > > to
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > incredibly picky about this public api
>> now
>> > so
>> > > > it
>> > > > > is
>> > > > > > > as
>> > > > > > > > > good
>> > > > > > > > > > > as
>> > > > > > > > > > > > > > > possible
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > we don't need to break it in the 
>>future.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > The best way to get a feel for the 
>>API is
>> > > > > actually
>> > > > > > to
>> > > > > > > > > take
>> > > > > > > > > > a
>> > > > > > > > > > > > look
>> > > > > > > > > > > > > > at
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > javadoc, my hope is to get the api 
>>docs
>> > good
>> > > > > enough
>> > > > > > > so
>> > > > > > > > > that
>> > > > > > > > > > > it
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > self-explanatory:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> 
>>http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/Kaf
>>kaProducer.html
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Please take a look at this API and 
>>give
>> me
>> > > any
>> > > > > > > thoughts
>> > > > > > > > > you
>> > > > > > > > > > > may
>> > > > > > > > > > > > > > have!
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > It may also be reasonable to take a 
>>look
>> at
>> > > the
>> > > > > > > > configs:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> 
>>http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerConfig
>>.html
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > The actual code is posted here:
>> > > > > > > > > > > > > > > > > >
>> > > > https://issues.apache.org/jira/browse/KAFKA-1227
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > A few questions or comments to kick
>> things
>> > > off:
>> > > > > > > > > > > > > > > > > > 1. We need to make a decision on 
>>whether
>> > > > > > > serialization
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > > > > > user's
>> > > > > > > > > > > > > > > > key
>> > > > > > > > > > > > > > > > > > and value should be done by the user
>> (with
>> > > our
>> > > > > api
>> > > > > > > just
>> > > > > > > > > > > taking
>> > > > > > > > > > > > > > > byte[])
>> > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > if we should take an object and allow 
>>the
>> > > user
>> > > > to
>> > > > > > > > > > configure a
>> > > > > > > > > > > > > > > > Serializer
>> > > > > > > > > > > > > > > > > > class which we instantiate via
>> reflection.
>> > We
>> > > > > take
>> > > > > > > the
>> > > > > > > > > > later
>> > > > > > > > > > > > > > approach
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the current producer, and I have 
>>carried
>> > this
>> > > > > > through
>> > > > > > > > to
>> > > > > > > > > > this
>> > > > > > > > > > > > > > > > prototype.
>> > > > > > > > > > > > > > > > > > The tradeoff I see is this: taking 
>>byte[]
>> > is
>> > > > > > actually
>> > > > > > > > > > > simpler,
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > user
>> > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > directly do whatever serialization 
>>they
>> > like.
>> > > > The
>> > > > > > > > > > > complication
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > actually
>> > > > > > > > > > > > > > > > > > partitioning. Currently partitioning 
>>is
>> > done
>> > > > by a
>> > > > > > > > similar
>> > > > > > > > > > > > plug-in
>> > > > > > > > > > > > > > api
>> > > > > > > > > > > > > > > > > > (Partitioner) which the user can
>> implement
>> > > and
>> > > > > > > > configure
>> > > > > > > > > to
>> > > > > > > > > > > > > > override
>> > > > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > > partitions are assigned. If we take
>> byte[]
>> > as
>> > > > > input
>> > > > > > > > then
>> > > > > > > > > we
>> > > > > > > > > > > > have
>> > > > > > > > > > > > > no
>> > > > > > > > > > > > > > > > > access
>> > > > > > > > > > > > > > > > > > to the original object and 
>>partitioning
>> > MUST
>> > > be
>> > > > > > done
>> > > > > > > on
>> > > > > > > > > the
>> > > > > > > > > > > > > byte[].
>> > > > > > > > > > > > > > > > This
>> > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > fine for hash partitioning. However 
>>for
>> > > various
>> > > > > > types
>> > > > > > > > of
>> > > > > > > > > > > > semantic
>> > > > > > > > > > > > > > > > > > partitioning (range partitioning, or
>> > > whatever)
>> > > > > you
>> > > > > > > > would
>> > > > > > > > > > want
>> > > > > > > > > > > > > > access
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > original object. In the current 
>>approach
>> a
>> > > > > producer
>> > > > > > > who
>> > > > > > > > > > > wishes
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > send
>> > > > > > > > > > > > > > > > > > byte[] they have serialized in their 
>>own
>> > code
>> > > > can
>> > > > > > > > > configure
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > BytesSerialization we supply which is
>> just
>> > a
>> > > > "no
>> > > > > > op"
>> > > > > > > > > > > > > serialization.
>> > > > > > > > > > > > > > > > > > 2. We should obsess over naming and 
>>make
>> > sure
>> > > > > each
>> > > > > > of
>> > > > > > > > the
>> > > > > > > > > > > class
>> > > > > > > > > > > > > > names
>> > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > good.
>> > > > > > > > > > > > > > > > > > 3. Jun has already pointed out that we
>> need
>> > > to
>> > > > > > > include
>> > > > > > > > > the
>> > > > > > > > > > > > topic
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > partition in the response, which is
>> > > absolutely
>> > > > > > > right. I
>> > > > > > > > > > > haven't
>> > > > > > > > > > > > > > done
>> > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > yet but that definitely needs to be
>> there.
>> > > > > > > > > > > > > > > > > > 4. Currently RecordSend.await will 
>>throw
>> an
>> > > > > > exception
>> > > > > > > > if
>> > > > > > > > > > the
>> > > > > > > > > > > > > > request
>> > > > > > > > > > > > > > > > > > failed. The intention here is that
>> > > > > > > > > > > > producer.send(message).await()
>> > > > > > > > > > > > > > > > exactly
>> > > > > > > > > > > > > > > > > > simulates a synchronous call. Guozhang
>> has
>> > > > noted
>> > > > > > that
>> > > > > > > > > this
>> > > > > > > > > > > is a
>> > > > > > > > > > > > > > > little
>> > > > > > > > > > > > > > > > > > annoying since the user must then 
>>catch
>> > > > > exceptions.
>> > > > > > > > > However
>> > > > > > > > > > > if
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > remove
>> > > > > > > > > > > > > > > > > > this then if the user doesn't check 
>>for
>> > > errors
>> > > > > they
>> > > > > > > > won't
>> > > > > > > > > > > know
>> > > > > > > > > > > > > one
>> > > > > > > > > > > > > > > has
>> > > > > > > > > > > > > > > > > > occurred, which I predict will be a
>> common
>> > > > > mistake.
>> > > > > > > > > > > > > > > > > > 5. Perhaps there is more we could do 
>>to
>> > make
>> > > > the
>> > > > > > > async
>> > > > > > > > > > > > callbacks
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > future
>> > > > > > > > > > > > > > > > > > we give back intuitive and easy to
>> program
>> > > > > against?
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Some background info on 
>>implementation:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > At a high level the primary 
>>difference in
>> > > this
>> > > > > > > producer
>> > > > > > > > > is
>> > > > > > > > > > > that
>> > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > removes
>> > > > > > > > > > > > > > > > > > the distinction between the "sync" and
>> > > "async"
>> > > > > > > > producer.
>> > > > > > > > > > > > > > Effectively
>> > > > > > > > > > > > > > > > all
>> > > > > > > > > > > > > > > > > > requests are sent asynchronously but
>> always
>> > > > > return
>> > > > > > a
>> > > > > > > > > future
>> > > > > > > > > > > > > > response
>> > > > > > > > > > > > > > > > > object
>> > > > > > > > > > > > > > > > > > that gives the offset as well as any
>> error
>> > > that
>> > > > > may
>> > > > > > > > have
>> > > > > > > > > > > > occurred
>> > > > > > > > > > > > > > > when
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > request is complete. The batching 
>>that is
>> > > done
>> > > > in
>> > > > > > the
>> > > > > > > > > async
>> > > > > > > > > > > > > > producer
>> > > > > > > > > > > > > > > > only
>> > > > > > > > > > > > > > > > > > today is done whenever possible now. 
>>This
>> > > means
>> > > > > > that
>> > > > > > > > the
>> > > > > > > > > > sync
>> > > > > > > > > > > > > > > producer,
>> > > > > > > > > > > > > > > > > > under load, can get performance as 
>>good
>> as
>> > > the
>> > > > > > async
>> > > > > > > > > > producer
>> > > > > > > > > > > > > > > > > (preliminary
>> > > > > > > > > > > > > > > > > > results show the producer getting 1m
>> > > > > messages/sec).
>> > > > > > > > This
>> > > > > > > > > > > works
>> > > > > > > > > > > > > > > similar
>> > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > group commit in databases but with
>> respect
>> > to
>> > > > the
>> > > > > > > > actual
>> > > > > > > > > > > > network
>> > > > > > > > > > > > > > > > > > transmission--any messages that arrive
>> > while
>> > > a
>> > > > > send
>> > > > > > > is
>> > > > > > > > in
>> > > > > > > > > > > > > progress
>> > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > batched together. It is also possible 
>>to
>> > > > > encourage
>> > > > > > > > > batching
>> > > > > > > > > > > > even
>> > > > > > > > > > > > > > > under
>> > > > > > > > > > > > > > > > > low
>> > > > > > > > > > > > > > > > > > load to save server resources by
>> > introducing
>> > > a
>> > > > > > delay
>> > > > > > > on
>> > > > > > > > > the
>> > > > > > > > > > > > send
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > allow
>> > > > > > > > > > > > > > > > > > more messages to accumulate; this is 
>>done
>> > > using
>> > > > > the
>> > > > > > > > > > > > > linger.msconfig
>> > > > > > > > > > > > > > > > > (this
>> > > > > > > > > > > > > > > > > > is similar to Nagle's algorithm in 
>>TCP).
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > This producer does all network
>> > communication
>> > > > > > > > > asynchronously
>> > > > > > > > > > > and
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > parallel
>> > > > > > > > > > > > > > > > > > to all servers so the performance 
>>penalty
>> > for
>> > > > > > acks=-1
>> > > > > > > > and
>> > > > > > > > > > > > waiting
>> > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > replication should be much reduced. I
>> > haven't
>> > > > > done
>> > > > > > > much
>> > > > > > > > > > > > > > benchmarking
>> > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > this yet, though.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > The high level design is described a
>> little
>> > > > here,
>> > > > > > > > though
>> > > > > > > > > > this
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > now
>> > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > little out of date:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > 
>>https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > -Jay
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>


Mime
View raw message