spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From akshay naidu <akshaynaid...@gmail.com>
Subject Run Multiple Spark jobs. Reduce Execution time.
Date Tue, 13 Feb 2018 11:13:56 GMT
Hello,
I'm try to run multiple spark jobs on cluster running in yarn.
Master is 24GB server with 6 Slaves of 12GB

fairscheduler.xml settings are -
<pool name="default">
    <schedulingMode>FAIR</schedulingMode>
    <weight>10</weight>
    <minShare>2</minShare>
</pool>

I am running 8 jobs simultaneously , jobs are running parallelly but not
all.
at a time only 7 of then runs simultaneously while the 8th one is in queue
WAITING for a job to stop.

also, out of the 7 running jobs, 4 runs comparatively much faster than
remaining three (maybe resources are not distributed properly) .

I want to run n number of jobs at a time and make them run faster , Right
now, one job is taking more than three minutes while processing a max of
1GB data .

Kindly assist me. what am I missing.

Thanks.

Mime
View raw message