sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ananth Durai <vanant...@gmail.com>
Subject Re: Custom output format / Avro-Parquet Support
Date Fri, 25 Apr 2014 22:43:01 GMT
Thanks Venkat, I gave it a try with HCatalog today. Since --as-avrodatafile
is depricated, I'm not able generate Avro object and give it to Parquet
Avro Output format. So the job not able to initialized.

This is my Hive table

create external table hcat_test (id BIGINT, name BIGINT)
  ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
  STORED AS
    INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
    OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
    LOCATION '/tmp/data/hcat_test';

my sqoop command

sqoop import --table test_db --username username --connect
"jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
--driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
--hcatalog-table hcat_employee


and I'm getting

14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
Exception in thread "main" java.lang.InstantiationError:
org.apache.hadoop.mapreduce.JobContext
at
parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
 at
org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
at
org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
 at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
 at
org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
 at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
 at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
 at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
 at org.apache.sqoop.Sqoop.main(Sqoop.java:238)




I suspect the absence of Avro Object causing the issue. Any suggestions pls.







Regards,
Ananth.P,







On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:

> Sqoop already is integrated with HCatalog so many different storage types
> should be possible to be used with Sqoop as long as there is a hive serde
> for it.   Check out the sqoop user guide on HCatalog integration.
>
>
> On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <vananth22@gmail.com> wrote:
>
>> How can I extend Sqoop client to support custom output format or provide
>> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
>> output format.
>>
>> Regards,
>> Ananth.P,
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <vananth22@gmail.com>wrote:
>>
>>> How can I extend Sqoop client to support custom output format or provide
>>> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
>>> output format.
>>>
>>> Regards,
>>> Ananth.P,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <vananth22@gmail.com>wrote:
>>>
>>>> How can I extend Sqoop client to support custom output format or
>>>> provide Avro-Parquet support? Right now Sqoop support only Text and
>>>> Sequence file output format.
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Ananth.P,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message