spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shay Seng <s...@1618labs.com>
Subject spark-ec2 launch script ... some issues and comments
Date Sat, 05 Oct 2013 01:00:55 GMT
Hi,

I've been trying to use the spark-ec2 launch scripts have have some
comments on it, not sure if this is the best place to post ...

(1) On the AMI image, most of the modeule's init.sh file has the following
idiom:
if [ -d "spark" ]; then
  echo "Spark seems to be installed. Exiting."
  exit 0
else
    # Github tag:
    if [[ "$SPARK_VERSION" == *\|* ]]
    then
...

I think the "exit 0" is an error - what it will do is cause the setup.sh
script which sources this init.sh script to also exit, causing the
bootstrap to stop at that point. This is certainly not what is wanted.

You can try see this by loading the AMI, installing spark by hand, and
using the new AMI as a source to spark-ec2.py

No other modules will be installed or setup.

(2) I want to push my AWS secret key-pair into the hadoop core-site.xml
file at startup. Currently I have to dive into the guts of the code and
perform surgery. It would be nice to have a more generic way to add
template variables etc.

Currently I have to modify the deploy_templates.py on the AMI to enable a
new template param.

Cheers,
shay

Mime
View raw message