sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Lin <eric....@cloudera.com>
Subject Re: Review Request 58466: SQOOP-3158 - Columns added to Mysql after initial sqoop import, export back to table with same schema fails
Date Sun, 21 May 2017 11:12:12 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58466/
-----------------------------------------------------------

(Updated May 21, 2017, 11:12 a.m.)


Review request for Sqoop, Attila Szabo and Szabolcs Vasas.


Changes
-------

update from hardcoded "null" value to this.options.getInNullStringValue() function call, and
also added a new test case.


Bugs: SQOOP-3158
    https://issues.apache.org/jira/browse/SQOOP-3158


Repository: sqoop-trunk


Description
-------

I have table in MySQL with 2 columns until yesterday. The columns are id and name.
1,Raj
2,Jack

I have imported this data into HDFS yesterday itself as a file. Today we added a new column
to the table in MySQL called salary. The table looks like below.

1,Raj
2,Jack
3,Jill,2000
4,Nick,3000

Now I have done Incremental import on this table as a file.

Part-m-00000 file contains
1,Raj
2,Jack

Part-m-00001 file contains
3,Jill,2000
4,Nick,3000

Now I created a new table in MySQL with same schema as Original MySQL table with columns id
name and salary.

Sqoop export will fail with below error:

java.lang.RuntimeException: Can't parse input data: 'Raj'
        at SQOOP_3158.__loadFromFields(SQOOP_3158.java:316)
        at SQOOP_3158.parse(SQOOP_3158.java:254)
        at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
        at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.util.NoSuchElementException
        at java.util.ArrayList$Itr.next(ArrayList.java:854)
        at SQOOP_3158.__loadFromFields(SQOOP_3158.java:311)
        ... 12 more


Diffs (updated)
-----

  src/java/org/apache/sqoop/orm/ClassWriter.java eaa9123 
  src/test/com/cloudera/sqoop/TestExport.java b2edc53 


Diff: https://reviews.apache.org/r/58466/diff/4/

Changes: https://reviews.apache.org/r/58466/diff/3-4/


Testing
-------

There is no existing test class to cover the path and I am not sure the best way to add test
case for this. If you have any suggestion, please let me know.

I have done manual testing to replicate the issue and confirmed that patch fixed the issue.
I have also tried different data types, all working.

However, if column in MySQL is defined as NOT NULL, then the export will still fail with error,
this is expected.


Thanks,

Eric Lin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message