spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From V Dineshkumar <developer.dines...@gmail.com>
Subject Spark SQL parallel query submission via single HiveContext
Date Mon, 29 Jun 2015 06:20:45 GMT
Hi,

As per my use case I need to submit multiple queries to Spark SQL in
parallel but due to HiveContext being thread safe the jobs are getting
submitted sequentially.
I could see many threads are waiting for HiveContext.

"on-spray-can-akka.actor.default-dispatcher-26" - Thread t@149
   java.lang.Thread.State: BLOCKED
at
org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:201)
*- waiting to lock <3b714d16> (a
org.apache.spark.sql.hive.HiveContext$$anon$2) owned by
"on-spray-can-akka.actor.default-dispatcher-14" t@117*


------------------------------------------------------------

"on-spray-can-akka.actor.default-dispatcher-16" - Thread t@122
   java.lang.Thread.State: BLOCKED
at
org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:201)
*- waiting to lock <3b714d16> (a
org.apache.spark.sql.hive.HiveContext$$anon$2) owned by
"on-spray-can-akka.actor.default-dispatcher-14" t@117*

I could use multiple HiveContext in different JVMs to submit the queries in
parallel but is there any other way I could achieve this using a *single
HiveContext*?

Thanks,
Dinesh.
Software Engineer,
Philips India

Mime
View raw message