sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandish Kumar HN <sanysand...@gmail.com>
Subject Re: Review Request 61522: SQOOP-2907 : Export parquet files to RDBMS: don't require .metadata for parquet files
Date Wed, 04 Oct 2017 07:24:18 GMT


> On Oct. 2, 2017, 3:18 p.m., Zoltán Tóth wrote:
> > src/java/org/apache/sqoop/avro/AvroUtil.java
> > Lines 194 (patched)
> > <https://reviews.apache.org/r/61522/diff/1/?file=1793959#file1793959line194>
> >
> >     If it is only Parquet file related then what is this change in AvroUtils. I
haven't checked why it is necessary so please explain why does this change needed.

bytes field in parquet throws that it can not cast BytesWriteble to String. so converting
from ByteWriteble to string with UTF-8


- Sandish Kumar


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61522/#review186844
-----------------------------------------------------------


On Oct. 4, 2017, 7:22 a.m., Sandish Kumar HN wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61522/
> -----------------------------------------------------------
> 
> (Updated Oct. 4, 2017, 7:22 a.m.)
> 
> 
> Review request for Sqoop and Anna Szonyi.
> 
> 
> Bugs: SQOOP-2907
>     https://issues.apache.org/jira/browse/SQOOP-2907
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> Kite currently requires .metadata.
> Parquet files have their own metadata stored along data files.
> It would be great for Export operation on parquet files to RDBMS not to require .metadata.
> We have most of the files created by Spark and Hive, and they don't create .metadata,
it only Kite that does.
> It makes sqoop export of parquet files usability very limited.
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/avro/AvroUtil.java ee29f140 
>   src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 6f9afaf9 
>   src/test/com/cloudera/sqoop/TestParquetExport.java 680fd73b 
> 
> 
> Diff: https://reviews.apache.org/r/61522/diff/2/
> 
> 
> Testing
> -------
> 
> testSupportedParquetTypesForWithoutParquetMeta - done
> testNullableFieldWithoutParquetMeta - done
> testParquetRecordsNotSupportedWithoutParquetMeta -done
> testMissingDatabaseFieldsWithoutParquetMeta - done
> testMissingParquetFieldsWithoutParquetMeta - done
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message