nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Jose Escobar <juanjose.esco...@gmail.com>
Subject Re: Nifi Benchmark/Performance tests
Date Tue, 03 Nov 2015 23:07:10 GMT
Hello, Mans,

Benchmarking can be tricky and results difficult to compare, specially for
Nifi because it may depend also on external systems (depending on the
processors you want to test), connectivity/bandwidth, etc.

For all internal interaction (fully controlled environment within a
cluster) I can confirm Joe's rough numbers. In a macbook pro I can easily
run my test flows at ~140MB/s (mainly receiving files via ListenHTTP, then
MergeContent and PutHDFS - with some UpdateAttibute, etc). You need to
tweak some config settings and configure multiple threads, etc in the
processors but nothing advanced. In the cluster this can go up to 800MB/s
using basic load balancing.

I noticed some improvement in 0.3.0, but I cannot guarantee that some other
changes in the rest of the flow did not affect. So I fully agree with Joe
that a set of specific benchmarks are needed as a reference. Those can also
double as blueprints for best practices and examples. I can contribute here.

J

On Tue, Nov 3, 2015 at 11:33 PM, Joe Witt <joe.witt@gmail.com> wrote:

> Mans,
>
> There aren't any published benchmarks but we really do need to do
> this.  We really would like to get access to a variety of system types
> to help with this.  But in the mean time some high level expectations
> for something like a raspberry pi would be thousands of events per
> second for a total of 10-20MB/s of throughput (with full provenance).
> On something like a macbook expect 10,000 events per second for 30+
> MB/s of throughput.  For something like a medium size server or
> something in EC2/Azure/etc.. with decent IOPs expect many 10s of
> thousands if not well north events/sec for again 40+MB/s of
> throughput.  Tricky bit here is that you can add multiple partitions
> for content and prov data which can also result in significant
> throughput gains.
>
> Thanks
> Joe
>
> On Tue, Nov 3, 2015 at 10:19 PM, M Singh <mans2singh@yahoo.com> wrote:
> > Hi:
> >
> > I wanted to find out if there are any performance benchmark
> > (latency/throughput) tests/stats available for nifi.
> >
> > Thanks
> >
> > Mans
>

Mime
View raw message