spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: How to increase number of Active Stages
Date Thu, 25 Sep 2014 06:43:05 GMT
Have a look at http://spark.apache.org/docs/1.0.0/tuning.html One thing you
can try is to increase the number of partition such as >= the number of
cores.

Thanks
Best Regards

On Thu, Sep 25, 2014 at 12:00 PM, Alexey Romanchuk <
alexey.romanchuk@gmail.com> wrote:

> Hello!
>
> I run local spark cluster with 64 cores total and perform data migration
> from protobuf to parquet. After consolidation number of protobuf files into
> one big parquet file I save it to hdfs and it takes a lot of time and uses
> only 1 core.
>
> To perform migration faster I start a lot of migration tasks in parallel.
> After some time I have all 8 active stages saving files and only 8 cores
> used. (See screenshot). Is there any way to increase the maximum number of
> active stages?
>
> Thanks
> [image: Inline image 2]
>

Mime
View raw message