spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weiwei Zhang <wzhan...@dons.usfca.edu>
Subject Configure Spark Resource on AWS CLI Not Working
Date Sat, 27 Feb 2016 02:37:59 GMT
Hi there,

I am trying to configure memory for spark using AWS CLI. However, I got the
following message:

*A client error (ValidationException) occurred when calling the RunJobFlow
operation: Cannot specify args for application 'Spark' when release label
is used.*

In the aws 'create-cluster' command, I have '--release-label emr-4.0.0
--applications Name=Hadoop
Name=Spark,Args=[-d,num-executors=4,spark.executor.memory=3000M,spark.driver.memory=4000M]'
and it seems like I cannot specify args when there is '--release-label'.
How do I get around this?

I also tried using a JSON configuration file saved in a S3 bucket and add
"--configurations http://path/bucket/config.json" to the command but it
gave me an 403 error (access denied). But when I did "aws s3 ls
(s3://bucket)" I could see that bucket and the config.json in the bucket.

Please advise. Thank you very much.

Best Regards,
Vivian

Mime
View raw message