tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Hive over Tez shuffle OOM
Date Sun, 26 Jan 2014 06:46:32 GMT
Hi all,

My Hive over Tez job fails with the following error while the same Hive
over MR  job succeeds, given the same input data and configurations.


*2014-01-26 13:46:19,823 ERROR [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hive (auth:SIMPLE)
cause:org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$ShuffleError:
error in shuffle in fetcher#115*
*    2014-01-26 13:46:19,824 FATAL [main]
org.apache.hadoop.mapred.YarnTezDagChild: Error running child :
org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$ShuffleError:
error in shuffle in fetcher#115*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$RunShuffleCallable.call(Shuffle.java:224)*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$RunShuffleCallable.call(Shuffle.java:204)*
*            at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)*
*            at java.util.concurrent.FutureTask.run(FutureTask.java:138)*
*            at java.lang.Thread.run(Thread.java:662)*
*    Caused by: java.lang.OutOfMemoryError: Java heap space*
*            at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)*
*            at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.MapOutput.<init>(MapOutput.java:98)*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.MergeManager.unconditionalReserve(MergeManager.java:295)*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.MergeManager.reserve(MergeManager.java:285)*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyMapOutput(Fetcher.java:395)*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyFromHost(Fetcher.java:325)*
*            at
org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.run(Fetcher.java:173)*

Is it merely a jvm configuration issue (like a larger heap memory) ?

Thanks,
Manu Zhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message