kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navneet Gupta (Tech - BLR)" <navneet.gu...@flipkart.com>
Subject Re: New Java Producer: Single Producer vs multiple Producers
Date Fri, 24 Apr 2015 18:42:05 GMT
Hi,

I ran some tests on our cluster by sending message from multiple clients
(machines). Each machine had about 40-100 threads per producer.

I thought of trying out having multiple producers per clients with each
producer receiving messages from say 10-15 threads. I actually did see an
increase in throughput in this case. It was not one off cases but a
repeatable phenomenon. I called threads to producer ratio sharingFactor in
my code.

I am not planning to use it this way in our clients sending messages to
Kafka but it did go against the suggestion to have single producer across
all threads.



On Fri, Apr 24, 2015 at 10:27 PM, Manikumar Reddy <kumar@nmsworks.co.in>
wrote:

> Hi Jay,
>
> Yes, we are producing from single process/jvm.
>
> From docs "The producer will attempt to batch records together into fewer
> requests whenever multiple records are being sent to the same partition."
>
> If I understand correctly, batching happens at topic/partition level, not
> at Node level. right?
>
> If yes, then  both (single producer for all topics , separate producer for
> each topic) approaches
> may give similar performance.
>
> On Fri, Apr 24, 2015 at 9:29 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
>
> > If you are talking about within a single process, having one producer is
> > generally the fastest because batching dramatically reduces the number of
> > requests (esp using the new java producer).
> > -Jay
> >
> > On Fri, Apr 24, 2015 at 4:54 AM, Manikumar Reddy <
> > manikumar.reddy@gmail.com>
> > wrote:
> >
> > > We have a 2 node cluster with 100 topics.
> > > should we use a single producer for all topics or  create multiple
> > > producers?
> > > What is the best choice w.r.t network load/failures, node failures,
> > > latency, locks?
> > >
> > > Regards,
> > > Manikumar
> > >
> >
>



-- 
Thanks & Regards,
Navneet Gupta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message