sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Spadotto <dougspado...@gmail.com>
Subject Re: Sqoop job to import data failing due to physical memory breach
Date Wed, 02 Aug 2017 13:49:54 GMT
Hello Harpreet,

It seems that your job is going beyond the limits established.

What are the values for yarn.scheduler.minimum-allocation-mb and
yarn.scheduler.maximum-allocation-mb
on your cluster?

Some background on the meaning of these configurations can be found here:
https://discuss.pivotal.io/hc/en-us/articles/201462036-MapReduce-YARN-Memory-Parameters

Regards,

Douglas

On Wed, Aug 2, 2017 at 8:00 AM, Harpreet Singh <hs.kundhal@gmail.com> wrote:

> Hi All,
> I have a sqoop job which is running in production and fails sometimes.
> Restart of job executes successfully .
> Logs show that failure happens with error that container is running beyond
> physical memory limits. Current usage 2.3 GB of 2GB physical memory used.
> 4.0 GB of 4.2 GB virtual memory used. Killing container.
> Environment is
> Cdh5.8.3
> Sqoop 1 client
> Mapreduce.map.Java.opts=-Djava.net.preferIPv4Stack=true -Xmx1717986918
> Mapreduce.map.memory.MB= 2GB
>
> Sqoop job details. Pulling data from netezza using 6 mappers and putting
> into parquet format on hdfs. Data processed is 14 GB. Splits seem to be
> even.
> Please provide your insights.
>
> Regards
> Harpreet Singh
>

Mime
View raw message