From user-return-3499-apmail-sqoop-user-archive=sqoop.apache.org@sqoop.apache.org Tue Aug 25 10:03:51 2015 Return-Path: X-Original-To: apmail-sqoop-user-archive@www.apache.org Delivered-To: apmail-sqoop-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7483018C53 for ; Tue, 25 Aug 2015 10:03:51 +0000 (UTC) Received: (qmail 81250 invoked by uid 500); 25 Aug 2015 10:03:48 -0000 Delivered-To: apmail-sqoop-user-archive@sqoop.apache.org Received: (qmail 81217 invoked by uid 500); 25 Aug 2015 10:03:48 -0000 Mailing-List: contact user-help@sqoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@sqoop.apache.org Delivered-To: mailing list user@sqoop.apache.org Received: (qmail 81207 invoked by uid 99); 25 Aug 2015 10:03:48 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2015 10:03:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id BCACD1AAB7C for ; Tue, 25 Aug 2015 10:03:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id fclMUsdca7_T for ; Tue, 25 Aug 2015 10:03:41 +0000 (UTC) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id C0A5342BB0 for ; Tue, 25 Aug 2015 10:03:40 +0000 (UTC) Received: by lbbsx3 with SMTP id sx3so96361375lbb.0 for ; Tue, 25 Aug 2015 03:03:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=hNwRT0wW4PpW8Jjm5N/gRqllkWeR1XtrtiSfwTWm0yU=; b=O41KCuEVXsfOJNgQsCC8GWv9/fWXDSZLjYke797M2Aa9uu9kIwSgzpExkzeqJXQpYt YyXFeJ8ZZjmk+BbzZmawy4QoMpvBzCf3M56ALujra4prjzW37LY2QKPYDLQ1hWTz2hMW vIr/AKIqZ9LllWb9R7RNC5pGqACatV7Sth9J9vSoopSFmg6fsZmie3rzPHTJI8QF5ztc QLPJlZMUGelav5xWa8OCJe0jePxgWAOLJMBFPl+dSJF857hiIv+d+nbOeIXAhb/zs+dd Zlh77EdnqiJz1YZUkpIOdbuvR6Dqc7bVY5EPCeGTRAw4hMkfZ5qR7DPpsw+uy6GFBRE7 AIHw== X-Received: by 10.112.210.6 with SMTP id mq6mr24656594lbc.83.1440497014380; Tue, 25 Aug 2015 03:03:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.114.186.67 with HTTP; Tue, 25 Aug 2015 03:03:15 -0700 (PDT) In-Reply-To: References: From: Mario Amatucci Date: Tue, 25 Aug 2015 12:03:15 +0200 Message-ID: Subject: Re: To: user@sqoop.apache.org Content-Type: multipart/alternative; boundary=001a11c3cc6ed7a010051e1fd8b3 --001a11c3cc6ed7a010051e1fd8b3 Content-Type: text/plain; charset=UTF-8 play around with java vm heap setting https://altiscale.zendesk.com/hc/en-us/articles/200801519-Configuring-Memory-for-Mappers-and-Reducers-in-Hadoop-2 *Kind Regards,*Mario Amatucci On 27 July 2015 at 03:17, Eric wrote: > Hi here! > > I tried to run a sqoop export command on Amazon EMR. I succeed running it > on a small dataset, but with a bigger one, I get this error on the cluster: > > Error: java.io.IOException: org.postgresql.util.PSQLException: Tried to > end inactive copy > at > org.apache.sqoop.mapreduce.postgresql.PostgreSQLCopyExportMapper.cleanup(PostgreSQLCopyExportMapper.java:145) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:160) > at > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170) > Caused by: org.postgresql.util.PSQLException: Tried to end inactive copy > at > org.postgresql.core.v3.QueryExecutorImpl.endCopy(QueryExecutorImpl.java:814) > at org.postgresql.core.v3.CopyInImpl.endCopy(CopyInImpl.java:59) > at > org.apache.sqoop.mapreduce.postgresql.PostgreSQLCopyExportMapper.cleanup(PostgreSQLCopyExportMapper.java:142) > ... 9 more > > > and here my command : ./sqoop export --direct --connect > jdbc:postgresql://xxxx --username xxxx --table updates_test --export-dir > s3://xxx -P --fields-terminated-by ',' --enclosed-by '\"' --null-string '' > --null-non-string '' > > > I was wondering if I missed anything , or if anyone here have any clue ? > > Thank you so much for your help. > Eric > --001a11c3cc6ed7a010051e1fd8b3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
play around with java vm heap setting
<= div class=3D"gmail_default" style=3D"font-family:verdana,sans-serif;color:r= gb(0,102,0)">https://alt= iscale.zendesk.com/hc/en-us/articles/200801519-Configuring-Memory-for-Mappe= rs-and-Reducers-in-Hadoop-2


Kind Regards,
Mario Amatucci


On 27 July 2015 at 03:17, Eric <eric.= khun.pro@gmail.com> wrote:
=
Hi here!

I tried to run a sqoop export command on = Amazon EMR. I succeed running it on a small dataset, but with a bigger one,= I get this error on the cluster:

Error: java.io.IO= Exception: org.postgresql.util.PSQLException: Tried to end inactive copy
=C2=A0 =C2=A0 at org.apache.sqoop.mapreduce.postgresql.PostgreSQLCo= pyExportMapper.cleanup(PostgreSQLCopyExportMapper.java:145)
=C2= =A0 =C2=A0 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:160)
=
=C2=A0 =C2=A0 at org.apache.sqoop.mapreduce.AutoProgressMapper.run(Aut= oProgressMapper.java:64)
=C2=A0 =C2=A0 at org.apache.hadoop.mapre= d.MapTask.runNewMapper(MapTask.java:773)
=C2=A0 =C2=A0 at org.apa= che.hadoop.mapred.MapTask.run(MapTask.java:341)
=C2=A0 =C2=A0 at = org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
=C2= =A0 =C2=A0 at java.security.AccessController.doPrivileged(Native Method)
=C2=A0 =C2=A0 at javax.security.auth.Subject.doAs(Subject.java:415)=
=C2=A0 =C2=A0 at org.apache.hadoop.security.UserGroupInformation= .doAs(UserGroupInformation.java:1548)
=C2=A0 =C2=A0 at org.apache= .hadoop.mapred.YarnChild.main(YarnChild.java:170)
Caused by: org.= postgresql.util.PSQLException: Tried to end inactive copy
=C2=A0 = =C2=A0 at org.postgresql.core.v3.QueryExecutorImpl.endCopy(QueryExecutorImp= l.java:814)
=C2=A0 =C2=A0 at org.postgresql.core.v3.CopyInImpl.en= dCopy(CopyInImpl.java:59)
=C2=A0 =C2=A0 at org.apache.sqoop.mapre= duce.postgresql.PostgreSQLCopyExportMapper.cleanup(PostgreSQLCopyExportMapp= er.java:142)
=C2=A0 =C2=A0 ... 9 more


and here my command := ./sqoop export --direct --connect jdbc:postgresql://xxxx --username xxxx -= -table updates_test --export-dir s3://xxx =C2=A0-P --fields-terminated-by &= #39;,' --enclosed-by '\"' --null-string '' --null-= non-string ''


I was wonderi= ng if I missed anything , or if anyone here have any clue ?
<= br>
Thank you so much for your help.
=
Eric

--001a11c3cc6ed7a010051e1fd8b3--