spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bozeman, Christopher" <>
Subject Re: Spark on EMR
Date Fri, 19 Jun 2015 19:54:08 GMT
You can use Spark 1.4 on EMR AMI 3.8.0 if you install Spark as a 3rd party application using
the bootstrap action directly without the native Spark inclusion with 1.3.1.  See

Refer to
to determine the version build to install (for example, build 1.4.0.b includes sparkR and
all builds include Kinesis).


From: Kelly, Jonathan
Sent: Wednesday, June 17, 2015 6:56 AM
To: Eugen Cepoi; Hideyoshi Maeda
Cc: ayan guha; kamatsuoka; user
Subject: Re: Spark on EMR

Yes, for now it is a wrapper around the old install-spark BA, but that will change soon. The
currently supported version in AMI 3.8.0 is 1.3.1, as 1.4.0 was released too late to include
it in AMI 3.8.0. Spark 1.4.0 support is coming soon though, of course. Unfortunately, though
install-spark is currently being used under the hood, passing "-v,1.4.0" in the options is
not supported.

Sent from Nine<>

From: Eugen Cepoi <<>>
Sent: Jun 17, 2015 6:37 AM
To: Hideyoshi Maeda
Cc: ayan guha;kamatsuoka;user
Subject: Re: Spark on EMR

It looks like it is a wrapper around
So basically adding an option -v,1.4.0.a should work.

2015-06-17 15:32 GMT+02:00 Hideyoshi Maeda <<>>:
Any ideas what version of Spark is underneath?

i.e. is it 1.4? and is SparkR supported on Amazon EMR?

On Wed, Jun 17, 2015 at 12:06 AM, ayan guha <<>>

That's great news. Can I assume spark on EMR supports kinesis to hbase pipeline?
On 17 Jun 2015 05:29, "kamatsuoka" <<>>
Spark is now officially supported on Amazon Elastic Map Reduce:

View this message in context:
Sent from the Apache Spark User List mailing list archive at<>.

To unsubscribe, e-mail:<>
For additional commands, e-mail:<>

View raw message