hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16626) S3A ITestRestrictedReadAccess fails
Date Thu, 03 Oct 2019 20:19:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943880#comment-16943880
] 

Steve Loughran commented on HADOOP-16626:
-----------------------------------------

OK. I have now learned something.

When you call Configuration.addResource() it reloads all configs, so
all settings you've previously cleared get set again.

And we force in the contract/s3a.xml settings, don't we?

I'm going to change how we load that file (which declares the expected FS behaviour in the
common tests). I'm going to make that load optional and only load it in those s3a contract
tests, not the other S3A tests.


(pause)
Actually, that's not enough! The first call to Filesystem.get() will force service discovery
of all filesystems, which will force their class instantiation, and then any class which forces
in a config (HDFS) triggers this.

{code}
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:43)
	  at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:123)
	  at java.lang.Class.forName0(Class.java:-1)
	  at java.lang.Class.forName(Class.java:348)
	  at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2603)
	  at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:96)
	  at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
	  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:106)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:102)
	  at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451)
	  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:355)
	  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:317)
	  at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1989)
	  at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:746)
	  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:696)
	  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:607)
	  at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:230)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource mapred-default.xml
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:44)
	  at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:123)
	  at java.lang.Class.forName0(Class.java:-1)
	  at java.lang.Class.forName(Class.java:348)
	  at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2603)
	  at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:96)
	  at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
	  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:106)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:102)
	  at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451)
	  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:355)
	  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:317)
	  at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1989)
	  at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:746)
	  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:696)
	  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:607)
	  at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:230)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource mapred-site.xml
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:45)
	  at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:123)
	  at java.lang.Class.forName0(Class.java:-1)
	  at java.lang.Class.forName(Class.java:348)
	  at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2603)
	  at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:96)
	  at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
	  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:106)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:102)
	  at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451)
	  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:355)
	  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:317)
	  at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1989)
	  at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:746)
	  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:696)
	  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:607)
	  at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:230)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource yarn-default.xml
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:46)
	  at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:123)
	  at java.lang.Class.forName0(Class.java:-1)
	  at java.lang.Class.forName(Class.java:348)
	  at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2603)
	  at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:96)
	  at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
	  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:106)
	  at org.apache.hadoop.security.Groups.<init>(Groups.java:102)
	  at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451)
	  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:355)
	  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:317)
	  at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1989)
	  at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:746)
	  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:696)
	  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:607)
	  at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:230)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource yarn-site.xml
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:37)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.<clinit>(DistributedFileSystem.java:146)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource hdfs-default.xml
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:38)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.<clinit>(DistributedFileSystem.java:146)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource hdfs-rbf-default.xml
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:39)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.<clinit>(DistributedFileSystem.java:146)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource hdfs-site.xml
Breakpoint reached
	  at org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:893)
	  at org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:40)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.<clinit>(DistributedFileSystem.java:146)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at java.lang.Class.newInstance(Class.java:442)
	  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	  at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3310)
	  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3355)
	  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3394)
	  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:500)
	  at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
	  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:178)
	  at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
	  at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.setup(ITestRestrictedReadAccess.java:233)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	  at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	  at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:-1)
	  at java.lang.Thread.run(Thread.java:748)
loading resource hdfs-rbf-site.xml
{code}

and ViewFS triggers UGI init, which loads a jobconf if present, so forces all of yarn and
MR. to an extend this is good: once it's happened, it's happened.
 
What to do? I'm going to force trigger a load of the local FS, before I go anywhere near configuration
of the S3A FS, so they are all loaded.  



> S3A ITestRestrictedReadAccess fails
> -----------------------------------
>
>                 Key: HADOOP-16626
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16626
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Siddharth Seth
>            Assignee: Steve Loughran
>            Priority: Major
>
> Just tried running the S3A test suite. Consistently seeing the following.
> Command used 
> {code}
> mvn -T 1C  verify -Dparallel-tests -DtestsThreadCount=12 -Ds3guard -Dauth -Ddynamo -Dtest=moo
-Dit.test=ITestRestrictedReadAccess
> {code}
> cc [~stevel@apache.org]
> {code}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 5.335 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
> testNoReadAccess[raw](org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess)  Time
elapsed: 2.841 s  <<< ERROR!
> java.nio.file.AccessDeniedException: test/testNoReadAccess-raw/noReadDir/emptyDir/: getFileStatus
on test/testNoReadAccess-raw/noReadDir/emptyDir/: com.amazonaws.services.s3.model.AmazonS3Exception:
Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: FE8B4D6F25648BCD;
S3 Extended Request ID: hgUHzFskU9CcEUT3DxgAkYcWLl6vFoa1k7qXX29cx1u3lpl7RVsWr5rp27/B8s5yjmWvvi6hVgk=),
S3 Extended Request ID: hgUHzFskU9CcEUT3DxgAkYcWLl6vFoa1k7qXX29cx1u3lpl7RVsWr5rp27/B8s5yjmWvvi6hVgk=:403
Forbidden
>         at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:244)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2777)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2705)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2589)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:2377)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$10(S3AFileSystem.java:2356)
>         at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:2356)
>         at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.checkBasicFileOperations(ITestRestrictedReadAccess.java:360)
>         at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.testNoReadAccess(ITestRestrictedReadAccess.java:282)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>         at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>         at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>         at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>         at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon
S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: FE8B4D6F25648BCD; S3 Extended
Request ID: hgUHzFskU9CcEUT3DxgAkYcWLl6vFoa1k7qXX29cx1u3lpl7RVsWr5rp27/B8s5yjmWvvi6hVgk=),
S3 Extended Request ID: hgUHzFskU9CcEUT3DxgAkYcWLl6vFoa1k7qXX29cx1u3lpl7RVsWr5rp27/B8s5yjmWvvi6hVgk=
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
>         at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
>         at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
>         at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
>         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920)
>         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866)
>         at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$5(S3AFileSystem.java:1682)
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:370)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1675)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1651)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2758)
>         ... 23 more
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message