hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang_intel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8373) OOM for a simple query with spark.master=local [Spark Branch]
Date Thu, 22 Dec 2016 06:52:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15769284#comment-15769284
] 

liyunzhang_intel commented on HIVE-8373:
----------------------------------------

[~xuefuz] and [~lirui]:  i have run spark.master=local  successfully to avoid "OOM" exception
by appending following in conf/hive-env.sh
{code}
 export HADOOP_OPTS="$HADOOP_OPTS -XX:MaxPermSize=128m"
{code}
Refer the [answer|http://stackoverflow.com/questions/34476195/why-does-creating-hivecontext-fail-with-java-lang-outofmemoryerror-permgen-spa]
on stackoverflow to solve the OOM exception.

Do we need to increase the value of "MaxPermSize" to make hos run successfully on local mode?
if yes, we should add this on wiki.

> OOM for a simple query with spark.master=local [Spark Branch]
> -------------------------------------------------------------
>
>                 Key: HIVE-8373
>                 URL: https://issues.apache.org/jira/browse/HIVE-8373
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Xuefu Zhang
>
> I have a straigh forward query to run in Spark local mode, but get an OOM even though
the data volumn is tiny:
> {code}
> Exception in thread "Spark Context Cleaner" 
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
"Spark Context Cleaner"
> Exception in thread "Executor task launch worker-1" 
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
"Executor task launch worker-1"
> Exception in thread "Keep-Alive-Timer" 
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
"Keep-Alive-Timer"
> Exception in thread "Driver Heartbeater" 
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
"Driver Heartbeater"
> {code}
> The query is:
> {code}
> select product_name, avg(item_price) as avg_price from product join item on item.product_pk=product.product_pk
group by product_name order by avg_price;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message