spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <lian.cs....@gmail.com>
Subject Re: thrift jdbc server probably running queries as hive query
Date Tue, 11 Nov 2014 02:59:55 GMT
Hey Sadhan,

I really don't think this is Spark log... Unlike Shark, Spark SQL 
doesn't even provide a Hive mode to let you execute queries against 
Hive. Would you please check whether there is an existing HiveServer2 
running there? Spark SQL HiveThriftServer2 is just a Spark port of 
HiveServer2, and they share the same default listening port. I guess the 
Thrift server didn't start successfully because the HiveServer2 occupied 
the port, and your Beeline session was probably linked against HiveServer2.

Cheng

On 11/11/14 8:29 AM, Sadhan Sood wrote:
> I was testing out the spark thrift jdbc server by running a simple 
> query in the beeline client. The spark itself is running on a yarn 
> cluster.
>
> However, when I run a query in beeline -> I see no running jobs in the 
> spark UI(completely empty) and the yarn UI seem to indicate that the 
> submitted query is being run as a map reduce job. This is probably 
> also being indicated from the spark logs but I am not completely sure:
>
> 2014-11-11 00:19:00,492 INFO  ql.Context 
> (Context.java:getMRScratchDir(267)) - New scratch dir is 
> hdfs://xxxxxxxx:9000/tmp/hive-ubuntu/hive_2014-11-11_00-19-00_367_3847629323646885865-1
>
> 2014-11-11 00:19:00,877 INFO  ql.Context 
> (Context.java:getMRScratchDir(267)) - New scratch dir is 
> hdfs://xxxxxxxx:9000/tmp/hive-ubuntu/hive_2014-11-11_00-19-00_367_3847629323646885865-2
>
> 2014-11-11 00:19:04,152 INFO  ql.Context 
> (Context.java:getMRScratchDir(267)) - New scratch dir is 
> hdfs://xxxxxxxx:9000/tmp/hive-ubuntu/hive_2014-11-11_00-19-00_367_3847629323646885865-2
>
> 2014-11-11 00:19:04,425 INFO Configuration.deprecation 
> (Configuration.java:warnOnceIfDeprecated(1009)) - 
> mapred.submit.replication is deprecated. Instead, use 
> mapreduce.client.submit.file.replication
>
> 2014-11-11 00:19:04,516 INFO  client.RMProxy 
> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager 
> at xxxxxxxx:8032
>
> 2014-11-11 00:19:04,607 INFO  client.RMProxy 
> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager 
> at xxxxxxxx:8032
>
> 2014-11-11 00:19:04,639 WARN mapreduce.JobSubmitter 
> (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line 
> option parsing not performed. Implement the Tool interface and execute 
> your application with ToolRunner to remedy this
>
> 2014-11-11 00:00:08,806 INFO  input.FileInputFormat 
> (FileInputFormat.java:listStatus(287)) - Total input paths to process 
> : 14912
>
> 2014-11-11 00:00:08,864 INFO  lzo.GPLNativeCodeLoader 
> (GPLNativeCodeLoader.java:<clinit>(34)) - Loaded native gpl library
>
> 2014-11-11 00:00:08,866 INFO  lzo.LzoCodec 
> (LzoCodec.java:<clinit>(76)) - Successfully loaded & initialized 
> native-lzo library [hadoop-lzo rev 
> 8e266e052e423af592871e2dfe09d54c03f6a0e8]
>
> 2014-11-11 00:00:09,873 INFO  input.CombineFileInputFormat 
> (CombineFileInputFormat.java:createSplits(413)) - DEBUG: Terminated 
> node allocation with : CompletedNodes: 1, size left: 194541317
>
> 2014-11-11 00:00:10,017 INFO  mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(396)) - number of splits:615
>
> 2014-11-11 00:00:10,095 INFO  mapreduce.JobSubmitter 
> (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: 
> job_1414084656759_0115
>
> 2014-11-11 00:00:10,241 INFO  impl.YarnClientImpl 
> (YarnClientImpl.java:submitApplication(167)) - Submitted application 
> application_1414084656759_0115
>
>
> It seems like the query is being run as a hive query instead of spark 
> query. The same query works fine when run from spark-sql cli.
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message