sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Ranganathan <vranganat...@hortonworks.com>
Subject Re: Sqoop import into a Hive table with 2 columns as partition keys
Date Thu, 13 Mar 2014 21:30:22 GMT
If your partition columns are dynamic, then using hcatalog import will be
enough.   The Sqoop HCatalog integration section in the Sqoop  user guide
has more information on the HCatalog integration (that I linked earlier).
The restriction is that you can't have more than one static partition key.


Venkat


On Wed, Mar 12, 2014 at 11:28 PM, Richa Sharma
<mailtorichasharma@gmail.com>wrote:

> Thanks for the inputs.
>
> In my case the partition columns are dynamic.. so I guess it will make
> sense to load it into 2 hops.
>
> Venkat, thanks for sharing the ppt on Hcatalog - I couldn't find much
> online so it is very helpful :-)
>
> Richa
>
>
> On Thu, Mar 13, 2014 at 11:33 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
>> In Sqoop 1.4.4 we have Hcatalog Integration.   This allows importing data
>> into hive partition tables with multiple partition keys (but only can be a
>> static partitioning key).    See
>> http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_sqoop_hcatalog_integration
>>
>> There is a presentation done at Hadoop World on Hcatalog integration that
>> you can refer to for more information
>>
>>
>> https://cwiki.apache.org/confluence/download/attachments/27361435/SqoopHCatIntegration-HadoopWorld2013.pptx
>>
>> Venkat
>>
>>
>> On Wed, Mar 12, 2014 at 9:19 PM, bejoy ks <bejoyks@gmail.com> wrote:
>>
>>> AFAIK. Sqoop 1x imports supports hive single column partitions.
>>>
>>> If it is multi column partitions you might have to take a two hop
>>> approach
>>> 1. Load the data into a single/non partitioned hive tabe
>>> 2. From the above staging table load to the final one.
>>>
>>>
>>> On Tue, Mar 11, 2014 at 7:23 AM, Richa Sharma <
>>> mailtorichasharma@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to run sqoop import into a Hive table partitioned on 2
>>>> columns (col1 string, col2 string).
>>>>
>>>> I looked at the documentation for both --hive-import and --hcatalog and
>>>> could not find any such option.
>>>>
>>>> It works successfully when table is partitioned on 1 column but not
>>>> when i increase number of partition columns.
>>>>
>>>> Has anyone tried it and knows how to achieve it ?
>>>>
>>>> Thanks,
>>>> Richa
>>>>
>>>
>>>
>>>
>>> --
>>> Regards
>>>        Bejoy
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message