spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Is It Feasible for Spark 1.1 Broadcast to Fully Utilize the Ethernet Card Throughput?
Date Fri, 09 Jan 2015 10:21:13 GMT
​You can try the following:

- Increase ​spark.akka.frameSize (default is 10MB)
- Try using torrentBroadcast

Thanks
Best Regards

On Fri, Jan 9, 2015 at 3:41 PM, Jun Yang <yangjunpro@gmail.com> wrote:

> Guys,
>
> I have a question regarding to Spark 1.1 broadcast implementation.
>
> In our pipeline, we have a large multi-class LR model, which is about 1GiB
> size.
> To employ the benefit of Spark parallelism, a natural thinking is to
> broadcast this model file to the worker node.
>
> However, it looks that broadcast performance is not quite good.
>
> During the process of broadcasting the model file, I just monitor the
> network card throughput of worker node, their
> recv/write throughput is just around 30~40 MiB( our server box is equipped
> with 100MiB ethernet card).
>
> Is this the real limitation of Spark 1.1 broadcast implementation? Or
> there may be some configuration or tricks
> that can help make Spark broadcast perform better.
>
> Thanks
>
>
>
> --
> yangjunpro@gmail.com
> http://hi.baidu.com/yjpro
>

Mime
View raw message