sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jordi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1226) --password-file option triggers FileSystemClosed exception at end of Oozie action
Date Wed, 10 Aug 2016 14:11:20 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415320#comment-15415320
] 

Jordi commented on SQOOP-1226:
------------------------------

The same issue happens for CryptoFileLoader.java class which extends FilePasswordLoader and
overrides method loadPassword.

Why haven't you applied the same fix to CryptoFileLoader class?

In my project we are getting the same error but caused by CryptoFileLoader.java because in
our case the password file also is encrypted.

We are using sqoop 1.4.6



> --password-file option triggers FileSystemClosed exception at end of Oozie action
> ---------------------------------------------------------------------------------
>
>                 Key: SQOOP-1226
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1226
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.3
>         Environment: Centos 6.2 + jdk-1.6.0_31-fcs.x86_64
>            Reporter: David Morel
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 1.4.5
>
>         Attachments: SQOOP-1226.patch, SQOOP-1226.patch
>
>
> When using the --password-file option, a Sqoop action running inside an Oozie workflow
will ERROR out at the very end, like so:
> {noformat}
> 2013-10-31 13:38:45,095 INFO org.apache.sqoop.hive.HiveImport: Hive import complete.
> 2013-10-31 13:38:45,098 INFO org.apache.sqoop.hive.HiveImport: Export directory is empty,
removing it.
> 2013-10-31 13:38:45,213 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing
logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-10-31 13:38:45,217 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:mapred (auth:SIMPLE) cause:java.io.IOException: Filesystem closed
> 2013-10-31 13:38:45,218 WARN org.apache.hadoop.mapred.Child: Error running child
> java.io.IOException: Filesystem closed
> 	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:565)
> 	at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:589)
> 	at java.io.FilterInputStream.close(FilterInputStream.java:155)
> 	at org.apache.hadoop.util.LineReader.close(LineReader.java:149)
> 	at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:243)
> 	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:222)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:421)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:262)
> 2013-10-31 13:38:45,234 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the
task
> {noformat}
> With the --password option, the job completes with no error. I believe the --password-file
option handling closes the FS which happens to be shared with the Oozie launcher, which can't
write to it anymore on completion. The solution I found was adding:
> {noformat}
>   <property>
>     <name>fs.hdfs.impl.disable.cache</name>
>     <value>true</value>
>   </property>
> {noformat}
> in the sqoop action definition in the oozie workflow, and that works, but isn't really
handy.
> Details are at https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pdsxiy5C_IY/OD8wR0rhHgMJ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message