samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Pan <nickpa...@gmail.com>
Subject Re: Samza job throughput much lower than Kafka throughput
Date Wed, 20 May 2015 20:51:41 GMT
Hi, George,

Could you share w/ us the code and configuration of your sample test job?
Thanks!

-Yi

On Wed, May 20, 2015 at 1:19 PM, George Li <gli@ca.ibm.com> wrote:

> Hi,
>
> We are evaluating Samza's performance, and our sample job with
> TestPerformanceTask is much slower than a program reading directly from
> Kafka.
>
> Scenario:
>
> * Cluster:
> 1 master node for Zookeeper and yarn.
> 3 Kafka broker nodes
> 3 yarn worker nodes
>
> * Kafka:
> Topic has only 1 partition. Average message size is around 100 byte.
> On a yarn worker node, run the performance test program from Kafka
> repository to read the topic. The throughput is about 400k messages/sec
>
> *Samza
> Run TestPerformanceTask from Samza repository with no output stream
> defined, and the throughput no more than 130k messages/sec
>
>
> How can I explain/fix this performance difference?
>
> What I have done so far:
>
> 1. Monitor yarn worker node resource usage.
> When the job is running, cpu and memory usage are never more than 5%
> except at the beginning of the run. No significant network and disk IO
> either
>
> 2. Monitor worker node network traffic
> Tcpdump shows an interesting pattern. The yarn worker node will fetch a
> block of data from the kafka broker, and after that, it will handshake
> with the same kafka broker once every 100 ms for 300 ms before fetching
> the next block.
>
> If I increase systems.kafka.samza.fetch.threshold to 500k, i.e., 10x the
> default settings, this handshake lasts about 3 seconds. If I set
> fetch.threshold to 250k, this idle period then becomes 1.5 sec. It seems
> kafka consumer greatly outpaced process() call
>
> 3. Check Samza metrics
> I do not see any excessive network calls to master or kafka broker, i.e.,
> no more than 30 calls/sec. However, process-ms and choose-ms are
> approaching 2ms
>
> Any input would be greatly appreciated.
>
> Thanks,
>
> George
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message