spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng, Hao" <>
Subject RE: Executing hive query from Spark code
Date Tue, 03 Mar 2015 06:55:34 GMT
After some investigation, I found the CDH assembly includes package like org\apache\spark\sql\hive,
but the hive jars are not included. It should work by adding the extra hive dependencies jars
during the runtime as Felix did.

From: Cheng, Hao []
Sent: Tuesday, March 3, 2015 2:19 PM
To: Felix C; Ted Yu; nitinkak001
Cc: user
Subject: RE: Executing hive query from Spark code

I am not so sure how Spark SQL compiled in CDH, but if didn’t specify the –Phive and –Phive-thriftserver
flags during the build, most likely it will not work if just by providing the Hive lib jars
later on.  For example, does the HiveContext class exist in the assembly jar?

I am also quite curious with that, any hint will be appreciated.

From: Felix C []
Sent: Tuesday, March 3, 2015 12:59 PM
To: Ted Yu; nitinkak001
Cc: user
Subject: Re: Executing hive query from Spark code

It should work in CDH without having to recompile.

--- Original Message ---

From: "Ted Yu" <<>>
Sent: March 2, 2015 1:35 PM
To: "nitinkak001" <<>>
Cc: "user" <<>>
Subject: Re: Executing hive query from Spark code
Here is snippet of dependency tree for spark-hive module:

[INFO] org.apache.spark:spark-hive_2.10:jar:1.3.0-SNAPSHOT
[INFO] +- org.spark-project.hive:hive-metastore:jar:0.13.1a:compile
[INFO] |  +- org.spark-project.hive:hive-shims:jar:0.13.1a:compile
[INFO] |  |  +- org.spark-project.hive.shims:hive-shims-common:jar:0.13.1a:compile
[INFO] |  |  +- org.spark-project.hive.shims:hive-shims-0.20:jar:0.13.1a:runtime
[INFO] |  |  +- org.spark-project.hive.shims:hive-shims-common-secure:jar:0.13.1a:compile
[INFO] |  |  +- org.spark-project.hive.shims:hive-shims-0.20S:jar:0.13.1a:runtime
[INFO] |  |  \- org.spark-project.hive.shims:hive-shims-0.23:jar:0.13.1a:runtime
[INFO] +- org.spark-project.hive:hive-exec:jar:0.13.1a:compile
[INFO] |  +- org.spark-project.hive:hive-ant:jar:0.13.1a:compile
[INFO] |  |  \- org.apache.velocity:velocity:jar:1.5:compile
[INFO] |  |     \- oro:oro:jar:2.0.8:compile
[INFO] |  +- org.spark-project.hive:hive-common:jar:0.13.1a:compile
[INFO] +- org.spark-project.hive:hive-serde:jar:0.13.1a:compile

bq. is there a way to have the hive support without updating the assembly

I don't think so.

On Mon, Mar 2, 2015 at 12:37 PM, nitinkak001 <<>>
I want to run Hive query inside Spark and use the RDDs generated from that
inside Spark. I read in the documentation

"/Hive support is enabled by adding the -Phive and -Phive-thriftserver flags
to Spark’s build. This command builds a new assembly jar that includes Hive.
Note that this Hive assembly jar must also be present on all of the worker
nodes, as they will need access to the Hive serialization and
deserialization libraries (SerDes) in order to access data stored in Hive./"

I just wanted to know what -Phive and -Phive-thriftserver flags really do
and is there a way to have the hive support without updating the assembly.
Does that flag add a hive support jar or something?

The reason I am asking is that I will be using Cloudera version of Spark in
future and I am not sure how to add the Hive support to that Spark

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:<>
For additional commands, e-mail:<>

View raw message