sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodrigo Matihara (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SQOOP-1306) Allow Sqoop to move files from different FileSystems on incremental import
Date Thu, 17 Apr 2014 13:27:21 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972924#comment-13972924
] 

Rodrigo Matihara edited comment on SQOOP-1306 at 4/17/14 1:26 PM:
------------------------------------------------------------------

Good Venkat Ranganathan, I will try Oozie too.
But, I steel thinking that could be a good improvement on Sqoop, allow the use of different
file systems to temp dir and target dir.

Supposing that I call this:
{code}sqoop import jdbc:mysql://xxxxx:3306/sqoop --username root --password root --table yyyyy
--target-dir s3n://<bucket-name>/sqoop --incremental append --check-column id --last-value
120{code}
I saw this bug https://issues.apache.org/jira/browse/SQOOP-1303, that allows a different file
system on target-dir, so this command is not working on sqoop-1.4.4, but will work on sqoop-1.4.5,
its great, cause Sqoop will allow use different FileSystems to target-dir.

But, if I call this:
{code}sqoop import jdbc:mysql://xxxxx:3306/sqoop --username root --password root --table yyyyy
--target-dir s3n://<bucket-name>/sqoop --incremental append --check-column id --last-value
120{code}
and supposing that * sqoop.test.import.rootDir=file:/tmp/sqoop/ *
I will have problems, cause Sqoop will try to rename temp files that exists on /tmp/sqoop/...
to s3n://<bucket-name>/sqoop/...

I'm proposing to Sqoop allow, when temp-dir and target-dir have different FileSystems, really
moves files from temp to target.
I have coded this on a sqoop version that I'm studying, and I could send the class to you
guys check.
I'm sorry about my english, I think that it is not helping our discussion, but my intentions
are just to help the community ;P


was (Author: rmatihara):
Good Venkat Ranganathan, I will try Oozie too.
But, I steel thinking that could be a good improvement on Sqoop, allow the use of different
file systems to temp dir and target dir.

Supposing that I call this:
{code}sqoop import jdbc:mysql://xxxxx:3306/sqoop --username root --password root --table yyyyy
--target-dir s3n://<bucket-name>/sqoop --incremental append --check-column id --last-value
120{code}
I saw this bug https://issues.apache.org/jira/browse/SQOOP-1303, that allows a different file
system on target-dir, so this command is not working on sqoop-1.4.4, but will work on sqoop-1.4.5,
its great, cause Sqoop will allow use different FileSystems to target-dir.

But, if I call this:
{code}sqoop import jdbc:mysql://xxxxx:3306/sqoop --username root --password root --table yyyyy
--target-dir s3n://<bucket-name>/sqoop --incremental append --check-column id --last-value
120{code}
and supposing that *sqoop.test.import.rootDir=file:/tmp/sqoop/*
I will have problems, cause Sqoop will try to rename temp files that exists on /tmp/sqoop/...
to s3n://<bucket-name>/sqoop/...

I'm proposing to Sqoop allow, when temp-dir and target-dir have different FileSystems, really
moves files from temp to target.
I have coded this on a sqoop version that I'm studying, and I could send the class to you
guys check.
I'm sorry about my english, I think that it is not helping our discussion, but my intentions
are just to help the community ;P

> Allow Sqoop to move files from different FileSystems on incremental import
> --------------------------------------------------------------------------
>
>                 Key: SQOOP-1306
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1306
>             Project: Sqoop
>          Issue Type: Improvement
>    Affects Versions: 1.4.4
>            Reporter: Rodrigo Matihara
>         Attachments: scenarios.png
>
>
> Sqoop doesn't allow to move files from different FileSystems executing an import (--append
or --incremental).
> Trying to import using a local temp-dir and a S3 target-dir, the operation is complete,
however the file is not created in S3 bucket and this warning is raised:
> WARN  - Cannot append files to target dir; no such directory: _sqoop/15151724000000436_31417_localhost.localdomain_<tablename>
> Looking into on source, I found that AppendUtils.java considers that tempDir and targetDir
are at the same Filesystem.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message