sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abraham Fine" <...@abrahamfine.com>
Subject Re: Review Request 42327: SQOOP-2788: Parquet support for HdfsConnector
Date Thu, 21 Jan 2016 19:07:39 GMT


> On Jan. 21, 2016, 3:30 a.m., Colin Ma wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/GenericHdfsWriter.java,
line 29
> > <https://reviews.apache.org/r/42327/diff/7/?file=1202469#file1202469line29>
> >
> >     Is it possible to pass the LoaderContext instead of Schema?
> >     Schema is only used for parquetWriter, if add more writer in the future, all
the specific parameters can be stored in LoaderContext.

im not sure if this is something we need to do yet. i generally like to avoid the context
objects because they add a layer of complexity. i would wait to see if this argument list
gets out of hand and then we could use it.


> On Jan. 21, 2016, 3:30 a.m., Colin Ma wrote:
> > connector/connector-hdfs/pom.xml, line 79
> > <https://reviews.apache.org/r/42327/diff/7/?file=1202464#file1202464line79>
> >
> >     The dependency can be add to pom.xml and the version can be added as ${parquet.version}.
> >     
> >     connector/connector-hdfs/pom.xml and test/pom.xml can just use the dependency
without version.

thanks!


> On Jan. 21, 2016, 3:30 a.m., Colin Ma wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java,
line 34
> > <https://reviews.apache.org/r/42327/diff/7/?file=1202465#file1202465line34>
> >
> >     For these org.apache.haddop.mapred.*, is it possible to use org.apache.haddop.mapreduce.*
?

thanks!


> On Jan. 21, 2016, 3:30 a.m., Colin Ma wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/AVROIntermediateDataFormat.java,
line 66
> > <https://reviews.apache.org/r/42327/diff/7/?file=1202475#file1202475line66>
> >
> >     It can be deleted.

thanks!


> On Jan. 21, 2016, 3:30 a.m., Colin Ma wrote:
> > test/src/test/java/org/apache/sqoop/integration/connector/hdfs/ParquetTest.java,
line 60
> > <https://reviews.apache.org/r/42327/diff/7/?file=1202477#file1202477line60>
> >
> >     This method should be only for fromParquetTest, move this into fromParquetTest
will be better.

thanks!


> On Jan. 21, 2016, 3:30 a.m., Colin Ma wrote:
> > test/src/test/java/org/apache/sqoop/integration/connector/hdfs/ParquetTest.java,
line 63
> > <https://reviews.apache.org/r/42327/diff/7/?file=1202477#file1202477line63>
> >
> >     This test is not in group "slow", is it necessary to add (alwaysRun = true)?

thanks!


- Abraham


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42327/#review115545
-----------------------------------------------------------


On Jan. 21, 2016, 7:06 p.m., Abraham Fine wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42327/
> -----------------------------------------------------------
> 
> (Updated Jan. 21, 2016, 7:06 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2788
>     https://issues.apache.org/jira/browse/SQOOP-2788
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> read and write parquet in hdfsconnector
> 
> 
> Diffs
> -----
> 
>   connector/connector-hdfs/pom.xml 599631418ca63cc43d645c1ee1e7a73dc824b313 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
a813c479a07d68e14ed49936f642e762e5b37437 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
774221aaf5c8cdb8d26ca108fae239598b42229b 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartition.java
644de60581faf90ceb2fcef8d3e0544067791fcc 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToFormat.java
27d121f529ecb4d5bd79e2b8c74ab8f7cc15fb10 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/GenericHdfsWriter.java
2ccccc4a94a582c8b47ccdefa523d1fd1632e627 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsParquetWriter.java
PRE-CREATION 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsSequenceWriter.java
75c2e7ef192d7d9628e622cc3c5ef176e33a73d0 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsTextWriter.java
78cf9732fdb89689b04d43e4af70ca5a43732dbf 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java
11fcef2a38209c79928f582cf8aa03e889247f22 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java
985149cbb0d28b55a19d17076d996364d7f2ae90 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/AVROIntermediateDataFormat.java
d78fa8b72ecfe62eeec240e01597e7f2a7e4dd76 
>   pom.xml cb8a973abc96af1de905cebd80d30177cbaf1cb4 
>   test/pom.xml 644a9c7dbc746d4a3268532bdcf0babd4faaafba 
>   test/src/test/java/org/apache/sqoop/integration/connector/hdfs/ParquetTest.java PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/42327/diff/
> 
> 
> Testing
> -------
> 
> integration tests pass
> 
> 
> Thanks,
> 
> Abraham Fine
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message