@Krishna Rao - Sequence file provides us the compression we need. Converting the data into a non-sequence file is not an option since it won't be scale.

@Venkat - The version of sqoop I am using is 1.4.3. Does HCatalog have to be installed separately or does it come as part of CDH4.3 ?


-Deepak



On Fri, Aug 16, 2013 at 10:14 AM, Venkat Ranganathan <vranganathan@hortonworks.com> wrote:
The HCatalog integration handles tables with sequence files - It is part of 1.4.4 - It handles both exports and imports 

Venkat


On Fri, Aug 16, 2013 at 9:32 AM, Krishna Rao <krishnanjrao@gmail.com> wrote:
I've run into this problem as well. I ended up copying the table into a non-sequenceFile table just so I could sqoop it out (something along the lines of CREATE TABLE nonSeqTbl LIKE seqTbl; INSERT OVERWRITE nonSeqTbl SELECT * FROM seqTbl;).

Is there a plan to allow sqoop-exporting of sequence file tables?

Krishna


On 16 August 2013 17:26, Abraham Elmahrek <abe@cloudera.com> wrote:
Ah I believe you're correct. Was this data imported with Sqoop? If so, does the table you're exporting to differ from the table you imported from?


On Thu, Aug 15, 2013 at 11:38 PM, Deepak Konidena <deepakkoni@gmail.com> wrote:
Does sqoop-export support --as-sequence option? I know sqoop-import does.


-Deepak



On Thu, Aug 15, 2013 at 11:34 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
Hey There,

I believe you're missing the --as-sequence directive!

-Abe


On Thu, Aug 15, 2013 at 7:16 PM, Deepak Konidena <deepakkoni@gmail.com> wrote:
Hi,

I have a sequence file with with both (key,value) as org.apache.hadoop.io.Text

I am trying to export the data into a mysql table with (key,value) mapped to (varchar, blob) since the value is pretty big. and I get the following error:

(command) - sqoop export -m "1" -connect "jdbc:mysql://<host>:3306/database" --username "sqoop" --password "sqooppwd" --table "tablename"  --export-dir "/path/to/sequencefile" --verbose

java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.LongWritable
    at org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:95)
    at org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:38)
    at org.apache.sqoop.mapreduce.CombineFileRecordReader.getCurrentKey(CombineFileRecordReader.java:79)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getCurrentKey(MapTask.java:461)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.getCurrentKey(MapContextImpl.java:66)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCurrentKey(WrappedMapper.java:75)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

The export works fine when I create a text file like so,

<key,value1,value2,value3>

and upload it to hdfs using -CopyFromLocal

But, its only with sequence files that it doesn't seem to work. Any thoughts?

Thanks,
Deepak







CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.