spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bozeman, Christopher" <bozem...@amazon.com>
Subject Re: Spark on EMR
Date Fri, 19 Jun 2015 19:54:08 GMT
You can use Spark 1.4 on EMR AMI 3.8.0 if you install Spark as a 3rd party application using
the bootstrap action directly without the native Spark inclusion with 1.3.1.  See https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark

Refer to https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/VersionInformation.md
to determine the version build to install (for example, build 1.4.0.b includes sparkR and
all builds include Kinesis).

-Christopher


From: Kelly, Jonathan
Sent: Wednesday, June 17, 2015 6:56 AM
To: Eugen Cepoi; Hideyoshi Maeda
Cc: ayan guha; kamatsuoka; user
Subject: Re: Spark on EMR

Yes, for now it is a wrapper around the old install-spark BA, but that will change soon. The
currently supported version in AMI 3.8.0 is 1.3.1, as 1.4.0 was released too late to include
it in AMI 3.8.0. Spark 1.4.0 support is coming soon though, of course. Unfortunately, though
install-spark is currently being used under the hood, passing "-v,1.4.0" in the options is
not supported.

Sent from Nine<http://www.9folders.com/>

From: Eugen Cepoi <cepoi.eugen@gmail.com<mailto:cepoi.eugen@gmail.com>>
Sent: Jun 17, 2015 6:37 AM
To: Hideyoshi Maeda
Cc: ayan guha;kamatsuoka;user
Subject: Re: Spark on EMR

It looks like it is a wrapper around https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark
So basically adding an option -v,1.4.0.a should work.

https://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-spark-configure.html

2015-06-17 15:32 GMT+02:00 Hideyoshi Maeda <hideyoshi.maeda@gmail.com<mailto:hideyoshi.maeda@gmail.com>>:
Any ideas what version of Spark is underneath?

i.e. is it 1.4? and is SparkR supported on Amazon EMR?

On Wed, Jun 17, 2015 at 12:06 AM, ayan guha <guha.ayan@gmail.com<mailto:guha.ayan@gmail.com>>
wrote:

That's great news. Can I assume spark on EMR supports kinesis to hbase pipeline?
On 17 Jun 2015 05:29, "kamatsuoka" <kenjim@gmail.com<mailto:kenjim@gmail.com>>
wrote:
Spark is now officially supported on Amazon Elastic Map Reduce:
http://aws.amazon.com/elasticmapreduce/details/spark/



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EMR-tp23343.html
Sent from the Apache Spark User List mailing list archive at Nabble.com<http://Nabble.com>.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>




Mime
View raw message