spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From curtkohler <>
Subject Problems launching 1.2.0-SNAPSHOT cluster with Hive support on EC2
Date Tue, 18 Nov 2014 21:01:01 GMT
I've developed a Spark application using the 1.2.0-SNAPSHOP branch that
leverages Spark Streaming and Hive and can run it locally with no problem (I
need some fixes in the 1.2.0 branch). I successfully launched my EC2 cluster
by specifying a git commit hash from the 1.2.0-SNAPSHOT branch as the
version (as documented in the scripts)  and was able to run non-Hive Spark
jobs against it.  However, when I try and deploy my Hive application to it,
I am getting an exception (java.lang.ArrayStoreException:
org.apache.spark.sql.execution.SparkStrategies$CommandStrategy?) creating my
HiveContext.  I suspect that after the spark-ec2 launch script causes the
source code clone of the git repo/commit point, the assembly isn't being
built with Hive support (e.g. a -Phive profile on the mvn build command as
described in building Spark with Maven in the documentation).

Is anyone aware of a way I can get Hive support added into my compiled
assembly on the master/slaves using the spark_ec scripts (or where it would
make sense to hack the existing scripts to get it working).



View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message