sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaomin Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3171) Import as parquet jobs failed randomly while multiple jobs concurrently importing into targets with same parent
Date Tue, 27 Mar 2018 04:42:00 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415010#comment-16415010
] 

Xiaomin Zhang commented on SQOOP-3171:
--------------------------------------

[~sanysandish@gmail.com] KITE-1155 is fixed upstream, however I don't see the fix version for
KITE-1155.  Can we just update the dependency for Sqoop to include the Kite fix?  Thanks.

> Import as parquet jobs failed randomly while multiple jobs concurrently importing into
targets with same parent
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-3171
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3171
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Xiaomin Zhang
>            Assignee: Sandish Kumar HN
>            Priority: Major
>
> Running multiple parquet import jobs concurrently into below target directories:
> hdfs://ns/path/dataset1
> hdfs://ns/path/dataset2
> In some cases, one of the sqoop job will be failed with below error:
> 17/03/19 08:21:21 INFO mapreduce.Job: Job job_1488289274600_188649 failed with state
FAILED due to: Job commit failed: org.kitesdk.data.DatasetIOException: Could not cleanly delete
path:hdfs://ns/path/.temp/job_1488289274600_188649
> at org.kitesdk.data.spi.filesystem.FileSystemUtil.cleanlyDelete(FileSystemUtil.java:239)
> at org.kitesdk.data.spi.filesystem.TemporaryFileSystemDatasetRepository.delete(TemporaryFileSystemDatasetRepository.java:61)
> at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$MergeOutputCommitter.commitJob(DatasetKeyOutputFormat.java:395)
> at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File hdfs://ns/path/.temp does not exist.
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:705)
> at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:106)
> at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:763)
> at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:759)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:759)
> at org.kitesdk.data.spi.filesystem.FileSystemUtil.cleanlyDelete(FileSystemUtil.java:226)
> This is due to:
> https://issues.cloudera.org/browse/KITE-1155



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message