spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kelly, Jonathan" <jonat...@amazon.com>
Subject Re: Spark on EMR
Date Wed, 17 Jun 2015 13:55:37 GMT
Yes, for now it is a wrapper around the old install-spark BA, but that will change soon. The
currently supported version in AMI 3.8.0 is 1.3.1, as 1.4.0 was released too late to include
it in AMI 3.8.0. Spark 1.4.0 support is coming soon though, of course. Unfortunately, though
install-spark is currently being used under the hood, passing "-v,1.4.0" in the options is
not supported.

Sent from Nine<http://www.9folders.com/>

From: Eugen Cepoi <cepoi.eugen@gmail.com>
Sent: Jun 17, 2015 6:37 AM
To: Hideyoshi Maeda
Cc: ayan guha;kamatsuoka;user
Subject: Re: Spark on EMR

It looks like it is a wrapper around https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark
So basically adding an option -v,1.4.0.a should work.

https://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-spark-configure.html

2015-06-17 15:32 GMT+02:00 Hideyoshi Maeda <hideyoshi.maeda@gmail.com<mailto:hideyoshi.maeda@gmail.com>>:
Any ideas what version of Spark is underneath?

i.e. is it 1.4? and is SparkR supported on Amazon EMR?

On Wed, Jun 17, 2015 at 12:06 AM, ayan guha <guha.ayan@gmail.com<mailto:guha.ayan@gmail.com>>
wrote:

That's great news. Can I assume spark on EMR supports kinesis to hbase pipeline?

On 17 Jun 2015 05:29, "kamatsuoka" <kenjim@gmail.com<mailto:kenjim@gmail.com>>
wrote:
Spark is now officially supported on Amazon Elastic Map Reduce:
http://aws.amazon.com/elasticmapreduce/details/spark/



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EMR-tp23343.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>




Mime
View raw message