sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joey Echeverria" <j...@cloudera.com>
Subject Review Request: SQOOP-318 Add support for splittable lzo files with Hive
Date Fri, 19 Aug 2011 18:49:06 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1597/
-----------------------------------------------------------

Review request for Sqoop.


Summary
-------

I added a check when generating the create table string to see if the LzopCodec is in use.
If it is, it outputs

STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"

at the end of the create table command, otherwise it outputs the standard

STORED AS TEXTFILE

I also added a call to the DistributedLzoIndexer before the data is imported into Hive.


This addresses bug SQOOP-318.
    https://issues.apache.org/jira/browse/SQOOP-318


Diffs
-----

  src/java/com/cloudera/sqoop/hive/HiveImport.java 36c17ba 
  src/java/com/cloudera/sqoop/hive/TableDefWriter.java 7dd9135 
  src/test/com/cloudera/sqoop/hive/TestTableDefWriter.java 43b755e 

Diff: https://reviews.apache.org/r/1597/diff


Testing
-------

It includes a test for the create table syntax. I manually tested calling the indexer. I'm
not sure how to automate that without making LZO required to build.


Thanks,

Joey


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message