spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: configure to run multiple tasks on a core
Date Wed, 26 Nov 2014 08:14:00 GMT
What about running, say, 2 executors per machine, each of which thinks
it should use all cores?

You can also multi-thread your map function manually, directly, within
your code, with careful use of a java.util.concurrent.Executor

On Wed, Nov 26, 2014 at 6:57 AM, yotto <> wrote:
> I'm running a spark-ec2 cluster.
> I have a map task that calls a specialized C++ external app. The app doesn't
> fully utilize the core as it needs to download/upload data as part of the
> task. Looking at the worker nodes, it appears that there is one task with my
> app running per core.
> I'd like to better utilize the cpu resources with the hope of increasing
> throughput by running multiple tasks (with my app) per core in parallel.
> I see there is a spark.task.cpus config setting with a default value of 1.
> It appears though that this is used to go the other way than what I am
> looking for.
> Is there a way where I can specify multiple tasks per core rather than
> multiple cores per task?
> thanks for any help.
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message