sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arvind Prabhakar <arv...@apache.org>
Subject Re: [sqoop-user] Dynamic Partition & Sqoop
Date Mon, 08 Aug 2011 19:10:16 GMT
Hi Bejoy,

The support for dynamic partitions is not there out of the box. If
this is required for your use case, I would suggest filing a new
feature request Jira in [2].

[2] https://issues.apache.org/jira/browse/SQOOP

Thanks,
Arvind

On Mon, Aug 8, 2011 at 11:12 AM,  <bejoyks@gmail.com> wrote:
> Hi Arvind
>        With this option is it feasible to accomplish dynamic partitions with a single
SQOOP command?
> It does well for populating a static partition with values. It'd be great if you just
brief on this implementation.
>
> ie with the step as mentioned, in Sqoop import we need to provide the where clause to
fetch the relevant data from db and load it to some pre partition defined hive table, where
the partitioned hive column is defined by '--hive-partition-key' and the value defined by
'--hive-partition-value'. With this approach. In order to implement DYNAMIC partitions we
need to analyze and identify the different values corresponding to the partition column of
hive from db table. If there are n different values identified then n different SQOOP imports
with appropriate where clauses  need to be fired . Is my understanding right?
>
> Thank You
>
> Regards
> Bejoy K S
>
> -----Original Message-----
> From: "arvind@cloudera.com" <arvind@cloudera.com>
> Date: Mon, 8 Aug 2011 10:00:20
> To: <sqoop-user@incubator.apache.org>
> Reply-To: sqoop-user@cloudera.org
> Subject: Re: [sqoop-user] Dynamic Partition & Sqoop
>
> [bcc:sqoop-user@cloudera.org (deprecated), to:
> sqoop-user@incubator.apache.org. Please continue the discussion on
> Apache mailing lists - for more information see [1]].
>
> Hi,
>
> Sqoop does support populating Hive partitions. The relevant command
> line arguments you need to supply are --hive-partition-key <key-name>,
> --hive-partition-value <value-string>. Please refer to the user-guide
> for more details.
>
> Thanks,
> Arvind
>
> [1] https://cwiki.apache.org/confluence/display/SQOOP/Mailing+Lists
>
> On Mon, Aug 8, 2011 at 5:17 AM, Cyberlord <kkrishnan88@gmail.com> wrote:
>> I’d like to know if Dynamic Partition is supported by Sqoop.
>>
>> For eg. If I have a database(RDBMS) Informix and want to import data
>> into Hive,
>> Will it be possible for me to import the data through sqoop into Hive
>> after creating dynamic partitions (supported by Hive) on-the-fly ?
>>
>> I possibly thought of loading the data into a temporary table (through
>> Sqoop)
>> and then load it into the required table through Hive interface.
>>
>> I wanted to know if I could migrate the data directly into Hive
>> through Sqoop while creating dynamic partitions in Hive without
>> using a temporary table.
>>
>> If it is possible, please let me know how to do it.
>>
>> Thanks :)
>>
>> --
>> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop
mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to
incubator-sqoop-user-subscribe@apache.org.
>>
>
> --
> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop
mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to
incubator-sqoop-user-subscribe@apache.org.
>

Mime
View raw message