sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-352) Export of avro data imported from database table (using sqoop import) fails on year
Date Fri, 02 Mar 2012 07:22:59 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220749#comment-13220749

jiraposter@reviews.apache.org commented on SQOOP-352:

This is an automatically generated e-mail. To reply, visit:

Thanks for the patch Bilung. The logic of the change is sound but I am concerned about backward


    This change along with the change to toHiveType() and introduction of toAvroType() breaks
backward compatibility. 

One way to address this would be to keep the old abstract API of ConnManager and introduce
new, non-abstract methods toJavaType(String columnName, int sqlType) which by default ignores
the column name and returns toJavaType(sqlType). This way, you stay backwards compatible and
built in connection managers can use the new functionality.

Another way that could possibly address this is by overwriting the getColumnTypesForRawQuery(String
stmt) in MySQLManager and mapping the YEAR type to SMALLINT. Not sure if it will work or not,
but is worth exploring.

- Arvind

On 2012-03-01 05:17:15, Bilung Lee wrote:
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3845/
bq.  -----------------------------------------------------------
bq.  (Updated 2012-03-01 05:17:15)
bq.  Review request for Sqoop, Arvind Prabhakar, Tom White, and Jarek Cecho.
bq.  Summary
bq.  -------
bq.  MySQL YEAR is mapped to java.sql.Date when importing to HDFS. On the way back an exception
is thrown because the date and month fields are truncated (not used) - this is the MySQL behaviour.
bq.  This addresses bug SQOOP-352.
bq.      https://issues.apache.org/jira/browse/SQOOP-352
bq.  Diffs
bq.  -----
bq.    ./src/java/org/apache/sqoop/hive/TableDefWriter.java 1293040 
bq.    ./src/java/org/apache/sqoop/manager/ConnManager.java 1293040 
bq.    ./src/java/org/apache/sqoop/manager/MySQLManager.java 1293040 
bq.    ./src/java/org/apache/sqoop/manager/OracleManager.java 1293040 
bq.    ./src/java/org/apache/sqoop/manager/SqlManager.java 1293040 
bq.    ./src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 1293040 
bq.    ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1293040 
bq.    ./src/java/org/apache/sqoop/orm/ClassWriter.java 1293040 
bq.    ./src/test/com/cloudera/sqoop/TestConnFactory.java 1293040 
bq.    ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1293040 
bq.  Diff: https://reviews.apache.org/r/3845/diff
bq.  Testing
bq.  -------
bq.  Thanks,
bq.  Bilung

> Export of avro data imported from database table (using sqoop import) fails on year
> -----------------------------------------------------------------------------------
>                 Key: SQOOP-352
>                 URL: https://issues.apache.org/jira/browse/SQOOP-352
>             Project: Sqoop
>          Issue Type: Bug
>          Components: connectors/mysql
>            Reporter: Tom White
>            Assignee: Bilung Lee
> Report from Wing Yew Poon:
> I use a MySQL database for testing sqoop. I created a table with a representative collection
of data types for the columns, excluding blob and text (and their different-sized variants).
> I run sqoop import --as-avrodatafile to import the table into hdfs.
> I create a second table with the same schema, and run sqoop export to export the avro
data to this second table. The export fails with
> {noformat}
> java.io.IOException: java.sql.SQLException: Data truncated for column 'yyyy' at row 1
> 	at com.cloudera.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:180)
> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:540)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:649)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.sql.SQLException: Data truncated for column 'yyyy' at row 1
> 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
> 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
> 	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3558)
> 	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3490)
> 	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
> 	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2109)
> 	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2648)
> 	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2077)
> 	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1356)
> 	at com.cloudera.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:232)
> {noformat}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message