sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin, Nick" <NiMar...@pssd.com>
Subject RE: sqoop import to hive being killed by resource manager
Date Fri, 13 Mar 2015 00:28:33 GMT
Adding @sqoop user ☺

From: Steve Howard [mailto:stevedhoward@gmail.com]
Sent: Thursday, March 12, 2015 8:24 PM
To: user@hive.apache.org
Subject: sqoop import to hive being killed by resource manager

Hi All,
We have not been able to get what is in the subject line to run.  This is on hive 0.14.  While
pulling a billion row table from Oracle using 12 splits on the primary key, each job continually
runs out of memory such as below...

15/03/13 00:22:23 INFO mapreduce.Job: Task Id : attempt_1426097251374_0011_m_000011_0, Status
: FAILED
Container [pid=27919,containerID=container_1426097251374_0011_01_000013] is running beyond
physical memory limits. Current usage: 513.5 MB of 512 MB physical memory used; 879.3 MB of
1.0 GB virtual memory used. Killing container.
Dump of the process-tree for container_1426097251374_0011_01_000013 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 28078 27919 27919 27919 (java) 63513 834 912551936 131129 /usr/jdk64/jdk1.7.0_45/bin/java
-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m
-Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp
-Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild
172.27.2.57 52335 attempt_1426097251374_0011_m_000011_0 13
        |- 27919 27917 27919 27919 (bash) 1 2 9424896 317 /bin/bash -c /usr/jdk64/jdk1.7.0_45/bin/java
-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m
-Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp
-Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild
172.27.2.57 52335 attempt_1426097251374_0011_m_000011_0 13 1>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stdout
2>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stderr

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
We have tried several different sizes for various switches, but the job always fails.
Is this simply a function of the data, or is there another issue?

Thanks,
Steve
Mime
View raw message