spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <>
Subject Re: Pushing Spark to 10Gb/s
Date Tue, 11 Aug 2015 09:04:20 GMT
Hi Starch,

It also depends on the applications behavior, some might not be properly
able to utilize the network. If you are using say Kafka, then one thing
that you should keep in mind is the Size of the individual message and the
number of partitions that you are having. The higher the message size and
higher number of partitions (in kafka) will utilize the network properly.
With this combination, we have operated few pipelines running at 10Gb/s (~
1GB/s ).

Best Regards

On Tue, Aug 11, 2015 at 12:24 AM, Starch, Michael D (398M) <> wrote:

> All,
> I am trying to get data moving in and out of spark at 10Gb/s. I currently
> have a very powerful cluster to work on, offering 40Gb/s inifiniband links
> so I believe the network pipe should be fast enough.
> Has anyone gotten spark operating at high data rates before? Any advice
> would be appreciated.
> -Michael Starch
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message