sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abraham Elmahrek" <...@cloudera.com>
Subject Re: Review Request 35859: SQOOP-1493: Add ability to import/export true decimal in Avro instead of serializing it to String
Date Mon, 13 Jul 2015 23:39:46 GMT


> On July 13, 2015, 3:18 p.m., Jarek Cecho wrote:
> > Overall the patch looks good. I have only one concern - I've tried to load generated
avro file into Hive and it seems that it doesn't work. Here is what I've did:
> > 
> > 1) Create MySQL table
> > 
> > CREATE TABLE `decimals` (
> >   `id` int(11) NOT NULL AUTO_INCREMENT,
> >   `dec1` decimal(3,2) DEFAULT NULL,
> >   PRIMARY KEY (`id`)
> > ) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
> > 
> > 2) Insert some data into it
> > 
> > insert into decimals values(2, 1.00);
> > insert into decimals values(1, 0.04);
> > insert into decimals values(3, 5.10);
> > insert into decimals values(4, 3.33);
> > 
> > 3) Import data with Sqoop
> > 
> > sqoop import --connect jdbc:mysql://mysql/sqoop --username sqoop --password sqoop
--table decimals --as-avrodatafile
> > 
> > 4) Manually create corresponding table in Hive
> > 
> > create table decimals(id int, dec decimal(3,2)) stored as avro;
> > 
> > 5) Load data
> > 
> > load data inpath '/user/root/decimals' overwrite into table decimals;
> > 
> > 6) And finally verify data:
> > 
> > hive> select * from decimals;
> > OK
> > 2	NULL
> > 1	NULL
> > 3	NULL
> > 4	NULL
> > Time taken: 0.45 seconds, Fetched: 4 row(s)
> > 
> > I would expect to see the proper decimal values, but I'm getting NULLs instead.
I'm wondering whether I'm doing something wrong or whether we do have a small bug somewhere
(that might not necessarily be on the Sqoop side).

Hmm Hive and Impala might not support "fixed" data types with logical type "decimal". Good
catch. I'll change this to bytes.


- Abraham


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35859/#review91470
-----------------------------------------------------------


On July 11, 2015, 12:28 a.m., Abraham Elmahrek wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35859/
> -----------------------------------------------------------
> 
> (Updated July 11, 2015, 12:28 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1493
>     https://issues.apache.org/jira/browse/SQOOP-1493
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> commit 8583077a33956cd2a3abc05f73172210c31994a9
> Author: Abraham Elmahrek <abe@apache.org>
> Date:   Fri Jun 19 05:45:25 2015 +0300
> 
>     SQOOP-1493: Add ability to import/export true decimal in Avro instead of serializing
it to String
> 
> :100644 100644 2e3d884... 628491c... M  ivy/libraries.properties
> :100644 100644 ee3cf62... 5970981... M  src/java/org/apache/sqoop/avro/AvroUtil.java
> :100644 100644 d9569c5... 5650079... M  src/java/org/apache/sqoop/manager/ConnManager.java
> :100644 100644 20f056a... 76c3458... M  src/java/org/apache/sqoop/mapreduce/AvroExportMapper.java
> :100644 100644 aed1e72... a4ac46e... M  src/java/org/apache/sqoop/mapreduce/AvroOutputFormat.java
> :100644 100644 ab263c1... b60ee42... M  src/java/org/apache/sqoop/mapreduce/GenericRecordExportMapper
> :100644 100644 2576673... 8490582... M  src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
> :100644 100644 1c6f7f4... aff8d08... M  src/java/org/apache/sqoop/orm/ClassWriter.java
> :100644 100644 663828c... 42e36ab... M  src/test/com/cloudera/sqoop/TestAvroExport.java
> :100644 100644 260e80a... 2f680c8... M  src/test/com/cloudera/sqoop/TestAvroImport.java
> 
> 
> Diffs
> -----
> 
>   LICENSE.txt c36c7ad 
>   ivy/libraries.properties 2e3d884 
>   src/java/org/apache/sqoop/avro/AvroUtil.java ee3cf62 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java e19c17b 
>   src/java/org/apache/sqoop/manager/ConnManager.java d9569c5 
>   src/java/org/apache/sqoop/mapreduce/AvroExportMapper.java 20f056a 
>   src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 0ea5ca4 
>   src/java/org/apache/sqoop/mapreduce/AvroOutputFormat.java aed1e72 
>   src/java/org/apache/sqoop/mapreduce/GenericRecordExportMapper.java ab263c1 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 2576673 
>   src/java/org/apache/sqoop/orm/ClassWriter.java 1c6f7f4 
>   src/test/com/cloudera/sqoop/TestAvroExport.java 663828c 
>   src/test/com/cloudera/sqoop/TestAvroImport.java 260e80a 
>   src/test/org/apache/sqoop/TestAvroImport.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/35859/diff/
> 
> 
> Testing
> -------
> 
> ant test && manual
> 
> 
> Thanks,
> 
> Abraham Elmahrek
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message