sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joey Echeverria" <j...@cloudera.com>
Subject Re: Review Request: SQOOP-318 Add support for splittable lzo files with Hive
Date Mon, 22 Aug 2011 23:01:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1597/
-----------------------------------------------------------

(Updated 2011-08-22 23:01:36.319406)


Review request for Sqoop.


Changes
-------

I added lzop to the CodecMap and modified the tests to reference the codec with the short
name. I added a blurb at the end of the Hive documentation describing the splitting you get
with the lzop codec. I also fixed the checkstyle issues.


Summary
-------

I added a check when generating the create table string to see if the LzopCodec is in use.
If it is, it outputs

STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"

at the end of the create table command, otherwise it outputs the standard

STORED AS TEXTFILE

I also added a call to the DistributedLzoIndexer before the data is imported into Hive.


This addresses bug SQOOP-318.
    https://issues.apache.org/jira/browse/SQOOP-318


Diffs (updated)
-----

  src/docs/user/hive.txt 059d7cb 
  src/java/com/cloudera/sqoop/hive/HiveImport.java 36c17ba 
  src/java/com/cloudera/sqoop/hive/TableDefWriter.java 7dd9135 
  src/java/com/cloudera/sqoop/io/CodecMap.java 8564164 
  src/test/com/cloudera/sqoop/hive/TestTableDefWriter.java 43b755e 

Diff: https://reviews.apache.org/r/1597/diff


Testing
-------

It includes a test for the create table syntax. I manually tested calling the indexer. I'm
not sure how to automate that without making LZO required to build.


Thanks,

Joey


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message