sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henrique Andrade (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1306) Allow Sqoop to move files from different FileSystems on incremental import
Date Thu, 24 Apr 2014 13:40:15 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979729#comment-13979729

Henrique Andrade commented on SQOOP-1306:

Jarek and Venkat,

I think the functionality from Sqoop is pretty clear and Rodrigo is not talking about adding
new functionalities or change current ones. The main point here, is cost. Right now the temp
files are being written to S3 that means that we have the cost to upload those files using
Internet connection and have several unnecessary read and write operations on S3. What Rodrigo
is proposing is to keep the temp files generated from Sqoop on the local temp directory and
then at the end of the process move the final file that to S3 as is doing right now.

The change that Rodrigo performed is just to have the option to define a local temp dir or
S3 temp dir. If the user defines a s3 temp dir, it will have the same activities that has
on the current code if the user defines and local temp dir then the temp files will be placed
on the local temp dir and at the end will be moved to S3.

Makes sense?

> Allow Sqoop to move files from different FileSystems on incremental import
> --------------------------------------------------------------------------
>                 Key: SQOOP-1306
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1306
>             Project: Sqoop
>          Issue Type: Improvement
>    Affects Versions: 1.4.4
>            Reporter: Rodrigo Matihara
>         Attachments: scenarios.png
> Sqoop doesn't allow to move files from different FileSystems executing an import (--append
or --incremental).
> Trying to import using a local temp-dir and a S3 target-dir, the operation is complete,
however the file is not created in S3 bucket and this warning is raised:
> WARN  - Cannot append files to target dir; no such directory: _sqoop/15151724000000436_31417_localhost.localdomain_<tablename>
> Looking into on source, I found that AppendUtils.java considers that tempDir and targetDir
are at the same Filesystem.

This message was sent by Atlassian JIRA

View raw message