sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Incremental import using sqoop2
Date Sat, 24 Oct 2015 13:30:15 GMT
What version are you using Adam?

The incremental import in JDBC connector has been added in 1.99.6 via SQOOP-1805. Quickly
looking through the code it seems that the "last value” field [1] don’t have specified
size which in turn defaults to -1. It’s interesting to me that integration tests with this
one are succeeding whereas real life use does not. Could you create a JIRA for it [2]? I’ll
take a deeper look.


1: https://github.com/apache/sqoop/blame/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/IncrementalRead.java#L34
2: https://issues.apache.org/jira/browse/SQOOP/

> On Oct 24, 2015, at 5:29 AM, Adam Konarski <akonarski@gmail.com> wrote:
> Hello
> I would like to import data from MySQL table into HDFS. I have everything configured
and I am able to create simple job in sqoop-shell that is copying data. However, I would like
to copy each time only new records, but I am not sure how to achieve this. When I create job
there is a parameter named "check column" and I have columns like ID or eventTimestamp that
seem to be suitable there. However, in such case I should enter also "last value" then. Do
I have to manage this last value by myself and each time create new job with new "last value"?
Why in such case create a job if it is used only once and then has to be recreated? Is it
not possible for Sqoop to manage this, by storing each time new "last value" and import only
new records? Moreover, why I have this error message when I enter anything as "last value":
"Size of input exceeds allowance for this input field. Maximal allowed size is -1"?
> Best regards, 
> Adam

View raw message