spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Pentreath <nick.pentre...@gmail.com>
Subject Re: Issue with Spark on EC2 using spark-ec2 script
Date Thu, 07 Aug 2014 11:39:49 GMT
Ryan, did you come right with this?

I've just ran into the same problem on a new 1.0.0 cluster I spun up. The
issue was that my app was not running against the Spark master, but in
local mode (a default setting in my app that was a throwback from 0.9.1 and
was overriding the spark defaults on the cluster). So it was trying to
create temp files in the spark.local.dir, as set in the Spark config -
 spark/conf/spark-env.sh has export
SPARK_LOCAL_DIRS="/mnt/spark,/mnt2/spark"

But these directories do not get created on the master by the spark-ec2
launch scripts, only on the slaves.

If you want to be able to run local mode jobs on the master, you can do
"mkdir /mnt/spark" and "mkdir /mnt2/spark" and try again.

Interestingly, if you try running the app without the --master local[4]
flag, it should run it against the cluster workers and should actually work
fine. Let me know if that is the case.

Not sure if this should be considered a bug in ec2 scripts or not - I guess
we'd probably want to be able to run local mode jobs on the master.

Another issue is that, straight out the box if anyone spins up an EC2
cluster and just runs an example using run-example, they will hit this
error since the examples run in local[] by default. So it was pretty
confusing for me for a while since spark-shell worked fine and launched on
the cluster, but run-example did not and hit this error...

N


On Fri, Aug 1, 2014 at 3:32 PM, Dean Wampler <deanwampler@gmail.com> wrote:

> It looked like you were running in standalone mode (master set to
> local[4]). That's how I ran it.
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
>
> On Thu, Jul 31, 2014 at 8:37 PM, ratabora <ratabora@gmail.com> wrote:
>
>> Hey Dean! Thanks!
>>
>> Did you try running this on a local environment or one generated by the
>> spark-ec2 script?
>>
>> The environment I am running on is a 4 data node 1 master spark cluster
>> generated by the spark-ec2 script. I haven't modified anything in the
>> environment except for adding data to the ephemeral hdfs.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-on-EC2-using-spark-ec2-script-tp11088p11117.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>

Mime
View raw message