spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Zhu <jun....@vungle.com.INVALID>
Subject Spark Thriftserver on yarn, sql submit take long time.
Date Tue, 04 Jun 2019 06:00:17 GMT
Hi ,
Running thrift server on yarn.
It's fast when beeline client send query to thrift server, but it take a
while(about 90s) to submit to yarn cluster.
>From Thrift server log:

> *19/06/04 05:48:27* DEBUG SparkSQLOperationManager: Created Operation for
> explain select count(*) from perf_as_reportads with
> session=org.apache.hive.service.cli.session.HiveSessionImpl@1f30fc84,
> runInBackground=true
> 19/06/04 05:48:27 INFO SparkExecuteStatementOperation: Running query '*explain
> select count(*) from perf_as_reportads*' with
> c63b765f-050a-412d-a817-45d5a990b59d
> 19/06/04 05:49:06 INFO ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V8
> 19/06/04 05:49:06 INFO SessionState: Created local directory:
> /tmp/384b2bd2-53fd-4300-a0dd-3be6590a6029_resources
> 19/06/04 05:49:06 INFO SessionState: Created HDFS directory:
> /tmp/spark/spark/384b2bd2-53fd-4300-a0dd-3be6590a6029
> 19/06/04 05:49:06 INFO SessionState: Created local directory:
> /tmp/spark/384b2bd2-53fd-4300-a0dd-3be6590a6029
> 19/06/04 05:49:06 INFO SessionState: Created HDFS directory:
> /tmp/spark/spark/384b2bd2-53fd-4300-a0dd-3be6590a6029/_tmp_space.db
> 19/06/04 05:49:06 INFO HiveSessionImpl: Operation log session directory is
> created: /tmp/spark/operation_logs/384b2bd2-53fd-4300-a0dd-3be6590a6029
> 19/06/04 05:50:06 INFO ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V8
> 19/06/04 05:50:06 INFO SessionState: Created local directory:
> /tmp/714c6377-5151-4574-969b-2c1cb2ed0d02_resources
> 19/06/04 05:50:06 INFO SessionState: Created HDFS directory:
> /tmp/spark/spark/714c6377-5151-4574-969b-2c1cb2ed0d02
> 19/06/04 05:50:06 INFO SessionState: Created local directory:
> /tmp/spark/714c6377-5151-4574-969b-2c1cb2ed0d02
> 19/06/04 05:50:06 INFO SessionState: Created HDFS directory:
> /tmp/spark/spark/714c6377-5151-4574-969b-2c1cb2ed0d02/_tmp_space.db
> 19/06/04 05:50:06 INFO HiveSessionImpl: Operation log session directory is
> created: /tmp/spark/operation_logs/714c6377-5151-4574-969b-2c1cb2ed0d02
> *19/06/04 05:50:15* DEBUG SparkExecuteStatementOperation: == Parsed
> Logical Plan ==
> ExplainCommand 'Project [unresolvedalias('count(1), None)], false, false,
> false
> == Analyzed Logical Plan ==
> plan: string
> ExplainCommand 'Project [unresolvedalias('count(1), None)], false, false,
> false
> == Optimized Logical Plan ==
> ExplainCommand 'Project [unresolvedalias('count(1), None)], false, false,
> false
> == Physical Plan ==
> Execute ExplainCommand
>    +- ExplainCommand 'Project [unresolvedalias('count(1), None)], false,
> false, false
> *19/06/04 05:50:15* INFO SparkExecuteStatementOperation: Result Schema:
> StructType(StructField(plan,StringType,true))


Had set thrift server miniresource(10 instance) and initresource(10) on
yarn.
Any thought? Any config issue may related?
-- 
[image: vshapesaqua11553186012.gif] <https://vungle.com/>   *Jun Zhu*
Sr. Engineer I, Data
+86 18565739171

[image: in1552694272.png] <https://www.linkedin.com/company/vungle>    [image:
fb1552694203.png] <https://facebook.com/vungle>      [image:
tw1552694330.png] <https://twitter.com/vungle>      [image:
ig1552694392.png] <https://www.instagram.com/vungle>
Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China

Mime
View raw message