sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodrigo Matihara (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1306) Allow Sqoop to move files from different FileSystems on incremental import
Date Wed, 16 Apr 2014 22:24:14 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Rodrigo Matihara updated SQOOP-1306:

    Attachment: scenarios.png

Jarek, I understand your point, but I think that distcp is not what I need.
I created a diagram (attached file) with 2 scenarios, that I think that you will understand
what I'm seeking for.
The first scenario is working perfect today, I can import data from a MySQL to HDFS/S3 creating
temp files on the target-dir.
The second scenario is what I need, import data from MySQL to HDFS/S3, creating temp files
locally and write final files to HDFS/S3.

> Allow Sqoop to move files from different FileSystems on incremental import
> --------------------------------------------------------------------------
>                 Key: SQOOP-1306
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1306
>             Project: Sqoop
>          Issue Type: Improvement
>    Affects Versions: 1.4.4
>            Reporter: Rodrigo Matihara
>         Attachments: scenarios.png
> Sqoop doesn't allow to move files from different FileSystems executing an import (--append
or --incremental).
> Trying to import using a local temp-dir and a S3 target-dir, the operation is complete,
however the file is not created in S3 bucket and this warning is raised:
> WARN  - Cannot append files to target dir; no such directory: _sqoop/15151724000000436_31417_localhost.localdomain_<tablename>
> Looking into on source, I found that AppendUtils.java considers that tempDir and targetDir
are at the same Filesystem.

This message was sent by Atlassian JIRA

View raw message