hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <>
Subject [jira] [Commented] (HIVE-15313) Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document
Date Thu, 01 Dec 2016 01:39:58 GMT


Rui Li commented on HIVE-15313:

Seems these two configs are useful in several ways :) I'm also looking at them in HIVE-15302.
My plan is to identify the minimum set of needed jars (good for performance the avoid conflicts)
and update the wiki.
We can also make code change to automatically set it if user hasn't.

> Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document
> -----------------------------------------------------------------------------------
>                 Key: HIVE-15313
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: liyunzhang_intel
>            Priority: Minor
>         Attachments: performance.improvement.after.set.spark.yarn.archive.PNG
> According to [wiki|],
run queries in HOS16 and HOS20 in yarn mode.
> Following table shows the difference in query time between HOS16 and HOS20.
> ||Version||Total time||Time for Jobs||Time for preparing jobs||
> |Spark16|51|39|12|
> |Spark20|54|40|14| 
>  HOS20 spends more time(2 secs) on preparing jobs than HOS16. After reviewing the source
code of spark, found that following point causes this:
> code:[Client#distribute|],
In spark20, if spark cannot find spark.yarn.archive and spark.yarn.jars in spark configuration
file, it will first copy all jars in $SPARK_HOME/jars to a tmp directory and upload the tmp
directory to distribute cache. Comparing [spark16|],

> In spark16, it searches spark-assembly*.jar and upload it to distribute cache.
> In spark20, it spends 2 more seconds to copy all jars in $SPARK_HOME/jar to a tmp directory
if we don't set "spark.yarn.archive" or "spark.yarn.jars".
> We can accelerate the startup of hive on spark 20 by settintg "spark.yarn.archive" or
> set "spark.yarn.archive":
> {code}
>  zip $SPARK_HOME/jars/*
> $ hadoop fs -copyFromLocal 
> $ echo "spark.yarn.archive=hdfs:///xxx:8020/" >> conf/spark-defaults.conf
> {code}
> set "spark.yarn.jars":
> {code}
> $ hadoop fs mkdir spark-2.0.0-bin-hadoop 
> $hadoop fs -copyFromLocal $SPARK_HOME/jars/* spark-2.0.0-bin-hadoop 
> $ echo "spark.yarn.jars=hdfs:///xxx:8020/spark-2.0.0-bin-hadoop/*" >> conf/spark-defaults.conf
> {code}
> Suggest to add this part in wiki.
> performance.improvement.after.set.spark.yarn.archive.PNG shows the detail performance
impovement after setting spark.yarn.archive in small queries.

This message was sent by Atlassian JIRA

View raw message