sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ananth Durai <vanant...@gmail.com>
Subject Re: Custom output format / Avro-Parquet Support
Date Mon, 05 May 2014 18:33:35 GMT
Jarek, Surprisingly --as-avrodatafile is not supported for HCat export.
When I tried I got error like,

HCatalog job is not compatible with  AVRO format option as-avrodatafile
option.

Regards,
Ananth.P,







On Sun, May 4, 2014 at 1:55 PM, Jarek Jarcec Cecho <jarcec@apache.org>wrote:

> To my best knowledge the argument --as-avrodatafile is not deprecated -
> why do you think so Ananth?
>
> Jarcec
>
> On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote:
> > Thanks Venkat, I gave it a try with HCatalog today. Since
> --as-avrodatafile
> > is depricated, I'm not able generate Avro object and give it to Parquet
> > Avro Output format. So the job not able to initialized.
> >
> > This is my Hive table
> >
> > create external table hcat_test (id BIGINT, name BIGINT)
> >   ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
> >   STORED AS
> >     INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
> >     OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
> >     LOCATION '/tmp/data/hcat_test';
> >
> > my sqoop command
> >
> > sqoop import --table test_db --username username --connect
> > "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
> > --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
> > --hcatalog-table hcat_employee
> >
> >
> > and I'm getting
> >
> > 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
> >
> hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
> > Exception in thread "main" java.lang.InstantiationError:
> > org.apache.hadoop.mapreduce.JobContext
> > at
> >
> parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
> >  at
> >
> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
> > at
> >
> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
> >  at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
> >  at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> >  at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
> >  at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
> >  at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> > at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> >  at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> > at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
> >  at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> >  at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >  at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> >  at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> >
> >
> >
> >
> > I suspect the absence of Avro Object causing the issue. Any suggestions
> pls.
> >
> >
> >
> >
> >
> >
> >
> > Regards,
> > Ananth.P,
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
> > vranganathan@hortonworks.com> wrote:
> >
> > > Sqoop already is integrated with HCatalog so many different storage
> types
> > > should be possible to be used with Sqoop as long as there is a hive
> serde
> > > for it.   Check out the sqoop user guide on HCatalog integration.
> > >
> > >
> > > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <vananth22@gmail.com>
> wrote:
> > >
> > >> How can I extend Sqoop client to support custom output format or
> provide
> > >> Avro-Parquet support? Right now Sqoop support only Text and Sequence
> file
> > >> output format.
> > >>
> > >> Regards,
> > >> Ananth.P,
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <vananth22@gmail.com
> >wrote:
> > >>
> > >>> How can I extend Sqoop client to support custom output format or
> provide
> > >>> Avro-Parquet support? Right now Sqoop support only Text and Sequence
> file
> > >>> output format.
> > >>>
> > >>> Regards,
> > >>> Ananth.P,
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <vananth22@gmail.com
> >wrote:
> > >>>
> > >>>> How can I extend Sqoop client to support custom output format or
> > >>>> provide Avro-Parquet support? Right now Sqoop support only Text
and
> > >>>> Sequence file output format.
> > >>>>
> > >>>>
> > >>>>
> > >>>> Regards,
> > >>>> Ananth.P,
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > > to which it is addressed and may contain information that is
> confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> immediately
> > > and delete it from your system. Thank You.
>

Mime
View raw message