nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From M Singh <mans2si...@yahoo.com>
Subject Re: Nifi Benchmark/Performance tests
Date Fri, 06 Nov 2015 03:59:42 GMT
Thanks Joe & Juan for your response and I look forward to examples/best practices guidelines.
Mans


     On Tuesday, November 3, 2015 3:07 PM, Juan Jose Escobar <juanjose.escobar@gmail.com>
wrote:
   

 Hello, Mans,
Benchmarking can be tricky and results difficult to compare, specially for Nifi because it
may depend also on external systems (depending on the processors you want to test), connectivity/bandwidth,
etc. 
For all internal interaction (fully controlled environment within a cluster) I can confirm
Joe's rough numbers. In a macbook pro I can easily run my test flows at ~140MB/s (mainly receiving
files via ListenHTTP, then MergeContent and PutHDFS - with some UpdateAttibute, etc). You
need to tweak some config settings and configure multiple threads, etc in the processors but
nothing advanced. In the cluster this can go up to 800MB/s using basic load balancing.
I noticed some improvement in 0.3.0, but I cannot guarantee that some other changes in the
rest of the flow did not affect. So I fully agree with Joe that a set of specific benchmarks
are needed as a reference. Those can also double as blueprints for best practices and examples.
I can contribute here.
J
On Tue, Nov 3, 2015 at 11:33 PM, Joe Witt <joe.witt@gmail.com> wrote:

Mans,

There aren't any published benchmarks but we really do need to do
this.  We really would like to get access to a variety of system types
to help with this.  But in the mean time some high level expectations
for something like a raspberry pi would be thousands of events per
second for a total of 10-20MB/s of throughput (with full provenance).
On something like a macbook expect 10,000 events per second for 30+
MB/s of throughput.  For something like a medium size server or
something in EC2/Azure/etc.. with decent IOPs expect many 10s of
thousands if not well north events/sec for again 40+MB/s of
throughput.  Tricky bit here is that you can add multiple partitions
for content and prov data which can also result in significant
throughput gains.

Thanks
Joe

On Tue, Nov 3, 2015 at 10:19 PM, M Singh <mans2singh@yahoo.com> wrote:
> Hi:
>
> I wanted to find out if there are any performance benchmark
> (latency/throughput) tests/stats available for nifi.
>
> Thanks
>
> Mans




  
Mime
View raw message