sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-891) Sqoop export from S3 to MySQL fails when S3 is not default filesystem.
Date Mon, 20 May 2013 16:31:23 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662113#comment-13662113

Hudson commented on SQOOP-891:

Integrated in Sqoop-ant-jdk-1.6-hadoop23 #841 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/841/])
    SQOOP-891: Sqoop export from S3 to MySQL fails when S3 is not default filesystem. (Revision

     Result = SUCCESS
jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=fd756a0c403e2d3c982096b165e53c2fefe8f31f
Files : 
* src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java

> Sqoop export from S3 to MySQL fails when S3 is not default filesystem.
> ----------------------------------------------------------------------
>                 Key: SQOOP-891
>                 URL: https://issues.apache.org/jira/browse/SQOOP-891
>             Project: Sqoop
>          Issue Type: Bug
>          Components: tools
>    Affects Versions: 1.4.1-incubating
>         Environment: CDH4.1.3 on Amazon EC2
>            Reporter: Jurgen Van Gael
>             Fix For: 1.4.4
> I recently tried to use sqoop to export a Hive table that lives on S3 into my MySQL server
(sqoop export --options-file config.txt --table _universe --export-dir s3n://key:secret@mybucket/universe
--input-fields-terminated-by '\0001' -m 1 --input-null-string '\\N' --input-null-non-string
'\\N'^C). My Sqoop runs on a CDH4 cluster on EC2. I was getting errors such as the following:
> 13/02/11 17:37:15 ERROR security.UserGroupInformation: PriviledgedActionException as:XXX
(auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /universe/000000_0.snappy
> 13/02/11 17:37:15 ERROR tool.ExportTool: Encountered IOException running export job:
java.io.FileNotFoundException: File does not exist: /universe/000000_0.snappy
> Since the files do exist on S3, I was reminded of getting the same errors when running
Hive queries against this table. The reason Hive was failing back then is because of a bug
in CombineFileInputFormat when using it against a non-default file system. These issues have
since been fixed in Hadoop:
> https://issues.apache.org/jira/browse/MAPREDUCE-1806
> https://issues.apache.org/jira/browse/MAPREDUCE-2704
> I believe Sqoop uses a version of CombineFileInputFormat but as far as I can tell from
the latest sources on GIT hasn't incorporated the above fixes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message