From user-return-952-apmail-sqoop-user-archive=sqoop.apache.org@sqoop.apache.org Mon Feb 4 20:35:06 2013 Return-Path: X-Original-To: apmail-sqoop-user-archive@www.apache.org Delivered-To: apmail-sqoop-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B245EECC5 for ; Mon, 4 Feb 2013 20:35:06 +0000 (UTC) Received: (qmail 90968 invoked by uid 500); 4 Feb 2013 20:35:06 -0000 Delivered-To: apmail-sqoop-user-archive@sqoop.apache.org Received: (qmail 90937 invoked by uid 500); 4 Feb 2013 20:35:06 -0000 Mailing-List: contact user-help@sqoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@sqoop.apache.org Delivered-To: mailing list user@sqoop.apache.org Received: (qmail 90929 invoked by uid 99); 4 Feb 2013 20:35:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Feb 2013 20:35:06 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [194.228.2.91] (HELO smtp-out3.iol.cz) (194.228.2.91) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Feb 2013 20:34:56 +0000 Received: from antivir5.iol.cz (unknown [192.168.30.212]) by smtp-out3.iol.cz (Postfix) with ESMTP id 3CE2738E16 for ; Mon, 4 Feb 2013 21:34:36 +0100 (CET) Received: from localhost (antivir5.iol.cz [127.0.0.1]) by antivir5.iol.cz (Postfix) with ESMTP id 2946F1E8034 for ; Mon, 4 Feb 2013 21:34:36 +0100 (CET) X-Virus-Scanned: amavisd-new at iol.cz Received: from antivir5.iol.cz ([127.0.0.1]) by localhost (antivir5.iol.cz [127.0.0.1]) (amavisd-new, port 10224) with LMTP id 0itULDcBDnJk for ; Mon, 4 Feb 2013 21:34:36 +0100 (CET) Received: from port9.iol.cz (unknown [192.168.30.99]) by antivir5.iol.cz (Postfix) with ESMTP id D96431E8039 for ; Mon, 4 Feb 2013 21:34:35 +0100 (CET) X-SBRS: None X-SBRS-none: None X-RECVLIST: MTA-OUT-IOL X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkkuADIaEFFBMsSC/2dsb2JhbABFhzixcIYuc4IfAQEFgQkIAxgJJQ8FNYg4u1eOKYMpA4hmjTiQUoMd X-IPAS-Result: AkkuADIaEFFBMsSC/2dsb2JhbABFhzixcIYuc4IfAQEFgQkIAxgJJQ8FNYg4u1eOKYMpA4hmjTiQUoMd Received: from connect-mtv.cloudera.com (HELO jarcec-thinkpad) ([65.50.196.130]) by port9.iol.cz with ESMTP; 04 Feb 2013 21:34:34 +0100 Date: Mon, 4 Feb 2013 12:34:31 -0800 From: Jarek Jarcec Cecho To: user@sqoop.apache.org Subject: Re: SQL Server sqoop export very slow Message-ID: <20130204203430.GH26349@jarcec-thinkpad> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BLwZsoChvJp3+clz" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Checked: Checked by ClamAV on apache.org --BLwZsoChvJp3+clz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Nitin, I'm afraid that Microsoft SQL Server connector do not support direct mode a= nd thus the --direct parameter should not make any difference. Based on my = personal experience the Microsoft's connector have comparable performance w= ith the build-in one and thus I'm not expecting any change with regards to = performance. It seems that in your case you are using three threads (mappers) writing da= ta simultaneously into database using JDBC interface. I would say that 1 ho= ur for 2 GB data set is quite much. I would suggest to taking a look on SQL= Server performance metrics during the export (IO, CPU, ...) to see where i= s the bottleneck. Jarcec On Mon, Feb 04, 2013 at 02:51:17PM -0500, Nitin kak wrote: > Hi All, >=20 > I am working on the sqoop export to SQL Server. It took like 1 hr to > export 2 GB of data. Three mappers had spawned. I am using Default SQL > Server Connector provided by cloudera(Couldn't use MS Sql Server Connector > because of a bug in export). I was using --direct clause. Doesnt seem to > have much effect. Any clues why its so slow? Would the performance improve > once I start using MS SQLServerConnector? >=20 > Regards, > Nitin --BLwZsoChvJp3+clz Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJREBtWAAoJEEk45sAabwqGLqgQANzBCav7VLnv79RDx4xZ1X2J NTUJURUvlXz9FfrIc/F7h7H7rA6Uk/FoKPgv5xfpKP+7TThE2VR7T9PuVTuxHSqx 7fL/C3gLi1EeZglmisAyI1g14a8sU++zf4KJJu/e3ODLOrEEaONGCy2iS4dz8ODV VxAakP7xms+xoNSAKkdXL0OyQuPpjLQoMr+FuGvIFAlX5iyqR9pWuOb5HwEb08Yg 9VdAEZ4z+GnnmsthGFCYUG6FYqTKTQS5QYJd2sUXv4Y3H4d/zMy0kBQcKXocahnM +tz/V5P4kPgPf37NJyTFOBRAxWOAIwIz6GE5ukOQNE7yA46S3xBsJGv2kZjz4lYh OrDEEgdC8BKgv+s0xXUafKZ9JJlpiOwkJb7a5Ptxo6/cfnInM2Uc6DSHibLkj9iS VaZw2d8lMykvEMy0x7xWFjRtDft8G4pv6uU1IT/VE5p37UftaUegrHv0IiSQofLg z0uAwGQKm6EbGBcVIXZeJab2TtQ0/WvaMK1H8ZEIPV9sVES1IOJ7GlKUOAKh1vio ZFv4pyvRHI/gjCaWE+vbLay0DLhBqNcK+0PcDw6DYGim7JkQx2XAfybyHEsIrejR TchhfMYVScdBc5j3incJU4VIBdJQqGE45r3Z68sAw9pxlU6108CmtWPy6MQEikpS VMF8f03asfVGQWuBh8aa =5QAK -----END PGP SIGNATURE----- --BLwZsoChvJp3+clz--