sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tanzir Musabbir <tmusab...@outlook.com>
Subject RE: Incremental Import for Sqoop2
Date Fri, 17 May 2013 03:35:27 GMT
I faced the same issue, I ended up by running a shell-action (through oozie) after each execution
of sqoop.

Inside my option parameter file I had something like this:

--whereID >= A and ID <= B

After each execution of sqoop, I run a shell script which modifies the "where" clause parameter
by the chunk size based on my need.
So for each sqoop run, my option parameter becomes:
run:1--whereID >= 1 and ID <= 20
run:2--whereID >= 21 and ID <= 40
run:3--whereID >= 41 and ID <= 60
where chunk size = 20.

It worked for me!

- Tanzir

Date: Wed, 15 May 2013 00:02:51 +0530
Subject: Re: Fwd: Incremental Import for Sqoop2
From: rj.vasanthkumar@gmail.com
To: user@sqoop.apache.org

Try using Boundary query or sql statement while creating import job for filtering records.

For example: Importing id greater than 100
Boundary query: select min(id),max(id) from table where id>100

Sorry if I am wrong.

On Tue, May 14, 2013 at 11:06 PM, Joanne Chan <jchan@shutterstock.com> wrote:

Hi Jarek,

Thanks for the speedy response!

Any idea what the timeline is for this feature?

I am trying to utilize sqoop2 to pull new/updated data from mysql to hdfs.

Do you know of any workarounds for sqoop2 to get this job done without having to use incremental

Appreciate your input!

On Tue, May 14, 2013 at 1:23 PM, Jarek Jarcec Cecho <jarcec@apache.org> wrote:

Hi sir,

Sqoop 2 currently can't do incremental imports, however implementing this feature is definitely
in our plan!


On Tue, May 14, 2013 at 01:14:32PM -0400, Joanne Chan wrote:

> Is incremental import available for sqoop2?  If not, is that part of the

> plan?


> I can't seem to find any good example of how to create import job with

> incremental parameter.


> I am on:

> client version:

>   Sqoop 2.0.0-SNAPSHOT revision c0a43d436fdd9293bf10e4b874d1002441c056b1

> server version:

>   Sqoop 2.0.0-SNAPSHOT revision c0a43d436fdd9293bf10e4b874d1002441c056b1

-- JChan

Vasanth kumar RJ
View raw message