sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Custom output format / Avro-Parquet Support
Date Sun, 04 May 2014 20:55:22 GMT
To my best knowledge the argument --as-avrodatafile is not deprecated - why do you think so
Ananth?

Jarcec

On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote:
> Thanks Venkat, I gave it a try with HCatalog today. Since --as-avrodatafile
> is depricated, I'm not able generate Avro object and give it to Parquet
> Avro Output format. So the job not able to initialized.
> 
> This is my Hive table
> 
> create external table hcat_test (id BIGINT, name BIGINT)
>   ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
>   STORED AS
>     INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
>     OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
>     LOCATION '/tmp/data/hcat_test';
> 
> my sqoop command
> 
> sqoop import --table test_db --username username --connect
> "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
> --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
> --hcatalog-table hcat_employee
> 
> 
> and I'm getting
> 
> 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
> hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
> Exception in thread "main" java.lang.InstantiationError:
> org.apache.hadoop.mapreduce.JobContext
> at
> parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
>  at
> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
> at
> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
>  at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>  at
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
>  at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
>  at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
>  at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>  at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>  at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> 
> 
> 
> 
> I suspect the absence of Avro Object causing the issue. Any suggestions pls.
> 
> 
> 
> 
> 
> 
> 
> Regards,
> Ananth.P,
> 
> 
> 
> 
> 
> 
> 
> On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
> 
> > Sqoop already is integrated with HCatalog so many different storage types
> > should be possible to be used with Sqoop as long as there is a hive serde
> > for it.   Check out the sqoop user guide on HCatalog integration.
> >
> >
> > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <vananth22@gmail.com> wrote:
> >
> >> How can I extend Sqoop client to support custom output format or provide
> >> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
> >> output format.
> >>
> >> Regards,
> >> Ananth.P,
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <vananth22@gmail.com>wrote:
> >>
> >>> How can I extend Sqoop client to support custom output format or provide
> >>> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
> >>> output format.
> >>>
> >>> Regards,
> >>> Ananth.P,
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <vananth22@gmail.com>wrote:
> >>>
> >>>> How can I extend Sqoop client to support custom output format or
> >>>> provide Avro-Parquet support? Right now Sqoop support only Text and
> >>>> Sequence file output format.
> >>>>
> >>>>
> >>>>
> >>>> Regards,
> >>>> Ananth.P,
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> > to which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender immediately
> > and delete it from your system. Thank You.

Mime
View raw message