sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bejoy KS" <bejo...@gmail.com>
Subject Re: Sqoop downloads split into chunks
Date Thu, 24 May 2012 07:11:34 GMT
Hi Brian

Use the where clause and num mappers together to specify the total data volume to be imported
at a time and how this load has to be distributed between tasks.

Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Brian Tran <brian@box.com>
Date: Thu, 24 May 2012 00:04:22 
To: <user@sqoop.apache.org>
Reply-To: user@sqoop.apache.org
Subject: Sqoop downloads split into chunks

Hi Sqoop gurus,

I currently use Sqoop to import from MySQL into HDFS.

Some of the tables that I import have become significantly larger to the
point that a full dump significantly slows down the host.

I would like to split the imports into smaller chunks, but limit the number
of chunks I download in parallel to avoid significant load on the server.

Is there anything in Sqoop that provides this functionality?

The closest thing I could find in the Sqoop user guide was the
--num-mappers option, but using it to download in smaller chunks would
increase the server load as all the chunks are downloaded in parallel.



View raw message