sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-443) Calling sqoop with hive import is not working multiple times due to kept output directory
Date Fri, 04 May 2012 06:38:50 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268172#comment-13268172
] 

Cheolsoo Park commented on SQOOP-443:
-------------------------------------

Hi Nemon,

I like your idea. In fact, I often find myself removing output dir, so having such an option
will be very useful.

That being said, I think that we should treat removing output dir in hive import differently
from removing output dir in general sqoop import. In hive import, output dir is no more than
temporary staging dir whereas it is a permanent destination in general sqoop import. It makes
sense to me to clean up temporary staging dir when the job was successful because data is
safely moved to hive.

So I think that it is better to handle your suggestion in a separate jira.

Thoughts?


                
> Calling sqoop with hive import is not working multiple times due to  kept output directory
> ------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-443
>                 URL: https://issues.apache.org/jira/browse/SQOOP-443
>             Project: Sqoop
>          Issue Type: Improvement
>    Affects Versions: 1.4.0-incubating, 1.4.1-incubating
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>            Priority: Minor
>         Attachments: SQOOP-443.patch
>
>
> Hive is not removing input directory when doing "LOAD DATA" command in all cases. This
input directory is actually sqoop's export directory. Because this directory is kept, calling
same sqoop command twice is failing on exception "org.apache.hadoop.mapred.FileAlreadyExistsException:
Output directory $table already exists".
> This issue might be easily overcome by manual directory removal, however it's putting
unnecessary burden on users. It's also complicating executing saved jobs as there is additional
script execution needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message