sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benyi Wang <bewang.t...@gmail.com>
Subject Re: Does sqoop 1/2 support import as Parquet file?
Date Wed, 18 Jun 2014 22:25:30 GMT
To my understanding, there is not a "HCatalog" service in Cloudera Manager,
and I don't have to install hcatalog using RPM.

I tried HCatalog using sqoop1, but could not write Parquet format. Here are
what I did:

1. hadoop fs -mkdir /tmp/action_t
2. hive> create external table action_t ( ...) stored as parquet location
'/tmp/action_t';
3. sqoop import --connect jdbc:teradata://teraserver/DATABASE=PDMPUBLIC
--username bwang --password xxx --table action --split-by actionid
--num-mappers 1 --hcatalog-table action_t --compress --compression-codec
org.apache.hadoop.io.compress.SnappyCodec -- --batch-size 1000

The problems are:
1. the job finished successfully, but the file in /tmp/action_t/_TEMP is in
text format.
2. If I use "--hcatalog-table action_text --create-hcatalog-table", the
data is not loaded into Hive.

Did I miss something?


On Tue, Jun 17, 2014 at 5:57 PM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:

> Yes.   Sqoop2 does not support HCatalog.
>
> BTW,  We are going to enhance the HCatalog integration on Sqoop1 to
> support all the Hive 0.13 datatypes.   Just cleaing up the code and
> adding tests.   Will be posting for review.   Will try to do a parquet
> test also.
>
> Venkat
>
>
> On Tue, Jun 17, 2014 at 5:55 PM, Benyi Wang <bewang.tech@gmail.com> wrote:
> > There is an open JIRA SQOOP-1159 Sqoop2: HCatalog Integration. Is this
> right
> > "Sqoop2 doesn't support HCatalog"?
> >
> >
> > On Tue, Jun 17, 2014 at 5:45 PM, Jarek Jarcec Cecho <jarcec@apache.org>
> > wrote:
> >>
> >> Not directly at the moment. But you should be able to use the HCatalog
> >> integration to import into Parquet?
> >>
> >> Jarcec
> >>
> >> On Tue, Jun 17, 2014 at 05:31:33PM -0700, Benyi Wang wrote:
> >> > I'm using CDH 5.0.2.
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
View raw message