spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "喜之郎" <251922...@qq.com>
Subject 回复: can not use udf in hivethriftserver2
Date Tue, 31 May 2016 06:36:21 GMT
hi,lalit sharma. I add jar to SPARK_CLASSPATH env variable.
But spark thriftserver can not start.


errors:
###
16/05/31 02:29:12 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Found both spark.executor.extraClassPath and SPARK_CLASSPATH.
Use only the former.
        at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$8.apply(SparkConf.scala:473)
        at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$8.apply(SparkConf.scala:471)
        at scala.collection.immutable.List.foreach(List.scala:318)







I found, udf can not work not because can not find jar. 
because when I create pernamant function, it says function have exists.
And in metastore, I found  a record:
8	com.dmp.hive.udfs.utils.URLEncode	1464623031	1	urlencode	1		USER.


I think It should be a privilege problem.But I can not make sure where the problem is ?
##########
logs:
org.apache.hive.service.cli.HiveSQLException: org.apache.spark.sql.AnalysisException: undefined
function URLEncode; line 1 pos 17
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.runInternal(SparkExecuteStatementOperation.scala:259)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:171)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:182)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)





------------------ 原始邮件 ------------------
发件人: "lalit sharma";<lalitkishore09@gmail.com>;
发送时间: 2016年5月31日(星期二) 凌晨2:15
收件人: "喜之郎"<251922566@qq.com>; 
抄送: "user"<user@spark.apache.org>; 
主题: Re: can not use udf in hivethriftserver2



Can you try adding jar to SPARK_CLASSPATH env variable ?


On Mon, May 30, 2016 at 9:55 PM, 喜之郎 <251922566@qq.com> wrote:
HI all, I have a problem when using hiveserver2 and beeline.
when I use CLI mode, the udf works well.
But when I begin to use hiveserver2 and beeline, the udf can not work.
My Spark version is 1.5.1.
I tried 2 methods, first:
######
add jar /home/hadoop/dmp-udf-0.0.1-SNAPSHOT.jar;
create temporary function URLEncode as "com.dmp.hive.udfs.utils.URLEncode" ;


errors:
Error: org.apache.spark.sql.AnalysisException: undefined function URLEncode; line 1 pos 207
(state=,code=0)




second:
create temporary function URLEncode as 'com.dmp.hive.udfs.utils.URLEncode' using jar 'hdfs:///warehouse/dmpv3.db/datafile/libjars/dmp-udf-0.0.1-SNAPSHOT.jar';


the error is same:
Error: org.apache.spark.sql.AnalysisException: undefined function URLEncode; line 1 pos 207
(state=,code=0)


###


can anyone give some suggestions? Or how to use udf in hiveserver2/beeline mode?
Mime
View raw message