spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishak <>
Subject Hive Session gets overwritten in ClientWrapper
Date Thu, 23 Jul 2015 05:31:09 GMT
I'm currently using Spark 1.4 in standalone mode.

I've forked the Apache Hive branch from
<>   and customised in the following way.

Added a thread local variable in SessionManager class. And I'm setting the
session variable in my Custom Authenticator class. 

     For achieving the above, I've built the necessary
jars(hive-common-0.13.1c.jar, hive-exec-0.13.1c.jar,
hive-metastore-0.13.1c.jar, hive-serde-0.13.1c.jar,
hive-service-0.13.1c.jar) from and added to
Spark's classpath. 
     The above feature works in Spark 1.3.1, but is broken in Spark 1.4.
When I looked into it, I found out that the ClientWrapper class is creating
a new Session State and using it thereafter. As a result I'm not able to
retrieve the info which i had stored earlier in the session. Also I'm not
able to retrieve a value from hiveconf which was set earlier.

      When i looked into the source code for ClientWrapper.scala, i found
the following.

       // Create an internal session state for this ClientWrapper.
  val state = {
    val original = Thread.currentThread().getContextClassLoader
    // Switch to the initClassLoader.

>From what i can understand, the above tries to use the existing Hive
session, else it creates it's own session. Am I right? If so, is there a bug
causing the ClientWrapper not to use the existing session. Or should I
implement my requirement in a different way?

My requirement is to have a custom session variable and use it throughout
the session.

My usage is as folows:

	To set the value

	To retrieve the value

	To set a hiveconf
	/hiveConf.set(conf, ConfVars.VAR, "val");/

	to Retrieve

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message