spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Romanchuk <alexey.romanc...@gmail.com>
Subject Re: How to increase number of Active Stages
Date Thu, 25 Sep 2014 06:58:02 GMT
Hey Akhil!

Thanks for reply. Yes, I have check docs from the official site. I need to
save exactly one partition and just want to increase number of active tasks.

On Thu, Sep 25, 2014 at 1:43 PM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> Have a look at http://spark.apache.org/docs/1.0.0/tuning.html One thing
> you can try is to increase the number of partition such as >= the number of
> cores.
>
> Thanks
> Best Regards
>
> On Thu, Sep 25, 2014 at 12:00 PM, Alexey Romanchuk <
> alexey.romanchuk@gmail.com> wrote:
>
>> Hello!
>>
>> I run local spark cluster with 64 cores total and perform data migration
>> from protobuf to parquet. After consolidation number of protobuf files into
>> one big parquet file I save it to hdfs and it takes a lot of time and uses
>> only 1 core.
>>
>> To perform migration faster I start a lot of migration tasks in parallel.
>> After some time I have all 8 active stages saving files and only 8 cores
>> used. (See screenshot). Is there any way to increase the maximum number of
>> active stages?
>>
>> Thanks
>> [image: Inline image 2]
>>
>
>

Mime
View raw message