sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Rao <krishnanj...@gmail.com>
Subject Re: Sqoop import when table schema changes
Date Tue, 28 Jan 2014 14:29:12 GMT
Thank you for your reply.

I think we'll probably just go for a simple solution: drop the table in
hive and then re-sqoop it in each time.


On 28 January 2014 14:17, Chalcy <chalcy@gmail.com> wrote:

> Yes, sqoop will not reflect schema changes.
>
> There are two ways you can handle,
> 1. Simple way is to add columns to the hive table and sqoop data into it.
>  In that case always the db columns need to be added to the end of the
> table.  (there is a way to handle if datatype of a column changed or column
> added in the middle)
> 2. If you have to handle schema changes on many tables, then you have to
> write a process to automatically find the schema changes and reflect that
> on the hive table.  This is a multi step process.
>
>
> On Tue, Jan 28, 2014 at 8:04 AM, Krishna Rao <krishnanjrao@gmail.com>wrote:
>
>> We have a task to sqoop-import tables from a postgreSQL db. With
>> "--hive-overwrite" the data correctly gets refreshed. However, if a schema
>> change occurs on the original table, it appears that the schema does not
>> get refreshed.
>>
>> Is there a way to get sqoop to re-create the table if the schema has
>> changed? Or will I need to delete the table beforehand?
>>
>> Krishna
>>
>
>

Mime
View raw message