spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shahab <shahab.mok...@gmail.com>
Subject Re: Supporting Hive features in Spark SQL Thrift JDBC server
Date Tue, 03 Mar 2015 16:49:16 GMT
Thanks Rohit,

I am already using Calliope and quite happy with it, well done ! except the
fact that :
1- It seems that it does not support Hive 0.12 or higher, Am i right?  for
example you can not use : current_time() UDF, or those new UDFs added in
hive 0.12 . Are they supported? Any plan for supporting them?
2-It does not support Spark 1.1 and 1.2. Any plan for new release?

best,
/Shahab

On Tue, Mar 3, 2015 at 5:41 PM, Rohit Rai <rohit@tuplejump.com> wrote:

> Hello Shahab,
>
> I think CassandraAwareHiveContext
> <https://github.com/tuplejump/calliope/blob/develop/sql/hive/src/main/scala/org/apache/spark/sql/hive/CassandraAwareHiveContext.scala>
in
> Calliopee is what you are looking for. Create CAHC instance and you should
> be able to run hive functions against the SchemaRDD you create from there.
>
> Cheers,
> Rohit
>
> *Founder & CEO, **Tuplejump, Inc.*
> ____________________________
> www.tuplejump.com
> *The Data Engineering Platform*
>
> On Tue, Mar 3, 2015 at 6:03 AM, Cheng, Hao <hao.cheng@intel.com> wrote:
>
>>  The temp table in metastore can not be shared cross SQLContext
>> instances, since HiveContext is a sub class of SQLContext (inherits all of
>> its functionality), why not using a single HiveContext globally? Is there
>> any specific requirement in your case that you need multiple
>> SQLContext/HiveContext?
>>
>>
>>
>> *From:* shahab [mailto:shahab.mokari@gmail.com]
>> *Sent:* Tuesday, March 3, 2015 9:46 PM
>>
>> *To:* Cheng, Hao
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: Supporting Hive features in Spark SQL Thrift JDBC server
>>
>>
>>
>> You are right ,  CassandraAwareSQLContext is subclass of SQL context.
>>
>>
>>
>> But I did another experiment, I queried Cassandra
>> using CassandraAwareSQLContext, then I registered the "rdd" as a temp table
>> , next I tried to query it using HiveContext, but it seems that hive
>> context can not see the registered table suing SQL context. Is this a
>> normal case?
>>
>>
>>
>> best,
>>
>> /Shahab
>>
>>
>>
>>
>>
>> On Tue, Mar 3, 2015 at 1:35 PM, Cheng, Hao <hao.cheng@intel.com> wrote:
>>
>>  Hive UDF are only applicable for HiveContext and its subclass instance,
>> is the CassandraAwareSQLContext a direct sub class of HiveContext or
>> SQLContext?
>>
>>
>>
>> *From:* shahab [mailto:shahab.mokari@gmail.com]
>> *Sent:* Tuesday, March 3, 2015 5:10 PM
>> *To:* Cheng, Hao
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: Supporting Hive features in Spark SQL Thrift JDBC server
>>
>>
>>
>>   val sc: SparkContext = new SparkContext(conf)
>>
>>   val sqlCassContext = new CassandraAwareSQLContext(sc)  // I used some
>> Calliope Cassandra Spark connector
>>
>> val rdd : SchemaRDD  = sqlCassContext.sql("select * from db.profile " )
>>
>> rdd.cache
>>
>> rdd.registerTempTable("profile")
>>
>>  rdd.first  //enforce caching
>>
>>      val q = "select  from_unixtime(floor(createdAt/1000)) from profile
>> where sampling_bucket=0 "
>>
>>      val rdd2 = rdd.sqlContext.sql(q )
>>
>>      println ("Result: " + rdd2.first)
>>
>>
>>
>> And I get the following  errors:
>>
>> xception in thread "main"
>> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved
>> attributes: 'from_unixtime('floor(('createdAt / 1000))) AS c0#7, tree:
>>
>> Project ['from_unixtime('floor(('createdAt / 1000))) AS c0#7]
>>
>>  Filter (sampling_bucket#10 = 0)
>>
>>   Subquery profile
>>
>>    Project
>> [company#8,bucket#9,sampling_bucket#10,profileid#11,createdat#12L,modifiedat#13L,version#14]
>>
>>     CassandraRelation localhost, 9042, 9160, normaldb_sampling, profile,
>> org.apache.spark.sql.CassandraAwareSQLContext@778b692d, None, None,
>> false, Some(Configuration: core-default.xml, core-site.xml,
>> mapred-default.xml, mapred-site.xml)
>>
>>
>>
>> at
>> org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$$anonfun$apply$1.applyOrElse(Analyzer.scala:72)
>>
>> at
>> org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$$anonfun$apply$1.applyOrElse(Analyzer.scala:70)
>>
>> at
>> org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:165)
>>
>> at
>> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:183)
>>
>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>
>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>
>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>
>> at
>> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>>
>> at
>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
>>
>> at
>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
>>
>> at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
>>
>> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
>>
>> at
>> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
>>
>> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
>>
>> at
>> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
>>
>> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
>>
>> at
>> org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:212)
>>
>> at
>> org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:168)
>>
>> at
>> org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:156)
>>
>> at
>> org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$.apply(Analyzer.scala:70)
>>
>> at
>> org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$.apply(Analyzer.scala:68)
>>
>> at
>> org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:61)
>>
>> at
>> org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:59)
>>
>> at
>> scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:51)
>>
>> at
>> scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:60)
>>
>> at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:34)
>>
>> at
>> org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:59)
>>
>> at
>> org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:51)
>>
>> at scala.collection.immutable.List.foreach(List.scala:318)
>>
>> at
>> org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:51)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:402)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:402)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.optimizedPlan$lzycompute(SQLContext.scala:403)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.optimizedPlan(SQLContext.scala:403)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:407)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:405)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:411)
>>
>> at
>> org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:411)
>>
>> at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:438)
>>
>> at org.apache.spark.sql.SchemaRDD.take(SchemaRDD.scala:440)
>>
>> at org.apache.spark.sql.SchemaRDD.take(SchemaRDD.scala:103)
>>
>> at org.apache.spark.rdd.RDD.first(RDD.scala:1091)
>>
>> at boot.SQLDemo$.main(SQLDemo.scala:65)  //my code
>>
>> at boot.SQLDemo.main(SQLDemo.scala)  //my code
>>
>>
>>
>> On Tue, Mar 3, 2015 at 8:57 AM, Cheng, Hao <hao.cheng@intel.com> wrote:
>>
>>  Can you provide the detailed failure call stack?
>>
>>
>>
>> *From:* shahab [mailto:shahab.mokari@gmail.com]
>> *Sent:* Tuesday, March 3, 2015 3:52 PM
>> *To:* user@spark.apache.org
>> *Subject:* Supporting Hive features in Spark SQL Thrift JDBC server
>>
>>
>>
>> Hi,
>>
>>
>>
>> According to Spark SQL documentation, "....Spark SQL supports the vast
>> majority of Hive features, such as  User Defined Functions( UDF) ", and one
>> of these UFDs is "current_date()" function, which should be supported.
>>
>>
>>
>> However, i get error when I am using this UDF in my SQL query. There are
>> couple of other UDFs which cause similar error.
>>
>>
>>
>> Am I missing something in my JDBC server ?
>>
>>
>>
>> /Shahab
>>
>>
>>
>>
>>
>
>

Mime
View raw message