sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Cecho" <jar...@apache.org>
Subject Re: Review Request 35859: SQOOP-1493: Add ability to import/export true decimal in Avro instead of serializing it to String
Date Mon, 13 Jul 2015 15:18:20 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35859/#review91470
-----------------------------------------------------------


Overall the patch looks good. I have only one concern - I've tried to load generated avro
file into Hive and it seems that it doesn't work. Here is what I've did:

1) Create MySQL table

CREATE TABLE `decimals` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `dec1` decimal(3,2) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;

2) Insert some data into it

insert into decimals values(2, 1.00);
insert into decimals values(1, 0.04);
insert into decimals values(3, 5.10);
insert into decimals values(4, 3.33);

3) Import data with Sqoop

sqoop import --connect jdbc:mysql://mysql/sqoop --username sqoop --password sqoop --table
decimals --as-avrodatafile

4) Manually create corresponding table in Hive

create table decimals(id int, dec decimal(3,2)) stored as avro;

5) Load data

load data inpath '/user/root/decimals' overwrite into table decimals;

6) And finally verify data:

hive> select * from decimals;
OK
2	NULL
1	NULL
3	NULL
4	NULL
Time taken: 0.45 seconds, Fetched: 4 row(s)

I would expect to see the proper decimal values, but I'm getting NULLs instead. I'm wondering
whether I'm doing something wrong or whether we do have a small bug somewhere (that might
not necessarily be on the Sqoop side).

- Jarek Cecho


On July 11, 2015, 12:28 a.m., Abraham Elmahrek wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35859/
> -----------------------------------------------------------
> 
> (Updated July 11, 2015, 12:28 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1493
>     https://issues.apache.org/jira/browse/SQOOP-1493
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> commit 8583077a33956cd2a3abc05f73172210c31994a9
> Author: Abraham Elmahrek <abe@apache.org>
> Date:   Fri Jun 19 05:45:25 2015 +0300
> 
>     SQOOP-1493: Add ability to import/export true decimal in Avro instead of serializing
it to String
> 
> :100644 100644 2e3d884... 628491c... M  ivy/libraries.properties
> :100644 100644 ee3cf62... 5970981... M  src/java/org/apache/sqoop/avro/AvroUtil.java
> :100644 100644 d9569c5... 5650079... M  src/java/org/apache/sqoop/manager/ConnManager.java
> :100644 100644 20f056a... 76c3458... M  src/java/org/apache/sqoop/mapreduce/AvroExportMapper.java
> :100644 100644 aed1e72... a4ac46e... M  src/java/org/apache/sqoop/mapreduce/AvroOutputFormat.java
> :100644 100644 ab263c1... b60ee42... M  src/java/org/apache/sqoop/mapreduce/GenericRecordExportMapper
> :100644 100644 2576673... 8490582... M  src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
> :100644 100644 1c6f7f4... aff8d08... M  src/java/org/apache/sqoop/orm/ClassWriter.java
> :100644 100644 663828c... 42e36ab... M  src/test/com/cloudera/sqoop/TestAvroExport.java
> :100644 100644 260e80a... 2f680c8... M  src/test/com/cloudera/sqoop/TestAvroImport.java
> 
> 
> Diffs
> -----
> 
>   LICENSE.txt c36c7ad 
>   ivy/libraries.properties 2e3d884 
>   src/java/org/apache/sqoop/avro/AvroUtil.java ee3cf62 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java e19c17b 
>   src/java/org/apache/sqoop/manager/ConnManager.java d9569c5 
>   src/java/org/apache/sqoop/mapreduce/AvroExportMapper.java 20f056a 
>   src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 0ea5ca4 
>   src/java/org/apache/sqoop/mapreduce/AvroOutputFormat.java aed1e72 
>   src/java/org/apache/sqoop/mapreduce/GenericRecordExportMapper.java ab263c1 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 2576673 
>   src/java/org/apache/sqoop/orm/ClassWriter.java 1c6f7f4 
>   src/test/com/cloudera/sqoop/TestAvroExport.java 663828c 
>   src/test/com/cloudera/sqoop/TestAvroImport.java 260e80a 
>   src/test/org/apache/sqoop/TestAvroImport.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/35859/diff/
> 
> 
> Testing
> -------
> 
> ant test && manual
> 
> 
> Thanks,
> 
> Abraham Elmahrek
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message