sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Ranganathan <vranganat...@hortonworks.com>
Subject Re: can I bulk import multiple mysql tables to one HBase/Hive table
Date Mon, 05 Aug 2013 21:43:14 GMT
Using the unions as Raghav says is the approach to take   I would suggest
doing the following

Create a query such as

    select name, id, address,... from (select user_name as name, user_id as
id, user_address as address ... union select customer_name as name,
customer_id as id,...union select employee_name as name,....) as tab

and then use sqoop to do the query-import

You can also pass some conditions in the where clause if needed.   Please
see the sqoop documentation on making either Hive or HBase


On Mon, Aug 5, 2013 at 2:06 PM, Raghav Gautam <rgautam@hortonworks.com>wrote:

> Hi Shengjie,
> You can try importing using --query and --boundary-query options.
> Your query might look something like:
> select * from A UNION select * from B UNION select * from C
> And you boundary queries might look like:
> select min(mintemp) from (select min(id) as mintemp from A UNION select
> min(id) as mintemp from B UNION select min(id) as mintemp from C) t;
> select max(maxtemp) from (select max(id) as maxtemp from A UNION select
> max(id) as maxtemp from B UNION select max(id) as maxtemp from C) t;
> If possible you can try creating a view and import that.
> With Regards,
> Raghav.
> On Mon, Aug 5, 2013 at 1:02 AM, Shengjie Min <kelvin.msj@gmail.com> wrote:
>> Hi guys,
>> I am reasonable new to Sqoop, just have a quick question here:
>> If I have multiple similar mysql tables, e.g:
>> table A: "users", columns: user_name, user_id, user_address, etc etc
>> table B: "customers" columns: customer_name, customer_id,
>> customer_address, etc etc
>> table C: "employee" columns: employee_name, employee_id,
>> employe_address, etc etc
>> Is it possible that using Sqoop to import the three tables into one HBase
>> or Hive table? So After the import, I have one HBase table contains all the
>> records in table A, B, C ?
>> --
>> All the best,
>> Shengjie Min

View raw message