ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhay Kulkarni <akulka...@hortonworks.com>
Subject Re: Review Request 70677: RANGER-2432: Upgrade Hadoop Version to 3.1.1
Date Fri, 24 May 2019 15:48:06 GMT


> On May 24, 2019, 3:18 p.m., Velmurugan Periasamy wrote:
> > Ship It!

About description of this JIRA: In my testing for hbase and hadoop up I found the following
to be the cause of the failure.

2019-05-23 16:23:56,805 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Formatting using clusterid: testClusterID
2019-05-23 16:23:58,033 WARN  [main] impl.MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2019-05-23 16:23:58,186 WARN  [main] http.HttpRequestLog: Jetty request log can only be enabled
using Log4j
2019-05-23 16:23:58,863 WARN  [main] common.MetricsLoggerTask: Metrics logging will not be
async since the logger is not log4j
2019-05-23 16:23:58,947 WARN  [main] http.HttpRequestLog: Jetty request log can only be enabled
using Log4j
2019-05-23 16:23:59,224 WARN  [main] common.MetricsLoggerTask: Metrics logging will not be
async since the logger is not log4j
2019-05-23 16:23:59,722 ERROR [main] server.ZooKeeperServer: ZKShutdownHandler is not registered,
so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2019-05-23 16:24:01,269 WARN  [Thread-107] hbase.ZNodeClearer: Environment variable HBASE_ZNODE_FILE
not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
2019-05-23 16:24:02,216 ERROR [Thread-107] config.RangerConfiguration: addResourceIfReadable(ranger-hbase-audit.xml):
couldn't find resource file location
2019-05-23 16:24:02,302 WARN  [RS:0;10.22.9.38:52417] regionserver.HRegionServer: reportForDuty
failed; sleeping and then retrying.
2019-05-23 16:24:02,916 WARN  [master/10.22.9.38:52415] assignment.AssignmentManager: No servers
available; cannot place 1 unassigned regions.
2019-05-23 16:24:03,933 WARN  [master/10.22.9.38:52415] assignment.AssignmentManager: No servers
available; cannot place 1 unassigned regions.
2019-05-23 16:24:04,946 WARN  [master/10.22.9.38:52415] assignment.AssignmentManager: No servers
available; cannot place 1 unassigned regions.
2019-05-23 16:24:05,324 WARN  [RS:0;10.22.9.38:52417] hbase.ZNodeClearer: Environment variable
HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
2019-05-23 16:24:08,521 ERROR [RS:0;10.22.9.38:52417] regionserver.HRegionServer: ***** ABORTING
region server 10.22.9.38,52417,1558653841129: Unhandled: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus,
but class was expected *****
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus,
but class was expected
	at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:768)
	at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:118)
	at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:848)
	at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:843)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:856)
	at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:168)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:165)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:124)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:764)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:494)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<init>(AsyncFSWAL.java:251)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:73)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:48)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
	at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:276)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2064)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1290)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1172)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:989)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164)
	at java.lang.Thread.run(Thread.java:748)
2019-05-23 16:24:08,524 ERROR [RS:0;10.22.9.38:52417] regionserver.HRegionServer: RegionServer
abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor]


Also, Hbase JIRA which addresses this is : https://issues.apache.org/jira/browse/HBASE-22394


- Abhay


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70677/#review215499
-----------------------------------------------------------


On May 19, 2019, 12:56 p.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70677/
> -----------------------------------------------------------
> 
> (Updated May 19, 2019, 12:56 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni,
Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and
Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2432
>     https://issues.apache.org/jira/browse/RANGER-2432
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **Following libraries can be upgraded:**
> 
> Hadoop version from 3.0.0 to 3.1.1
> HBase version from 2.0.0 to 2.0.2
> Hive version from 3.0.0 to 3.1.0
> Atlas version from 1.1.0 to 2.0.0
> 
> 
> **Note:** Disabled Hbase test cases as setup() in HBaseRangerAuthorizationTest is failing.
It could be due to any of the issues mentioned in https://issues.apache.org/jira/browse/HBASE-21688.
If its so, then probably fix is not available in the released hbase versions as I have tried
all released Hbase > 2.0.0. Also have tried with all the combination of hadoop > 3.0.0
> It appears this issue is due to changes in hadoop after 3.0.0 release but not fixed from
hbase end in released version. 
> 
> I could not find any work around to get them working as earlier. If you have any approach
to fix these testcases please leave your comment here or in the Jira. 
> 
> **Stacktrace after the testcase failure:**
> 
> Thread 1 (main):
>   State: RUNNABLE
>   Blocked count: 50
>   Waited count: 499
>   Stack:
>     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
>     org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:169)
>     sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     java.lang.reflect.Method.invoke(Method.java:498)
>     org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294)
>     org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341)
>     org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221)
>     org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
>     org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:258)
>     org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:115)
>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1038)
>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:984)
>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:855)
>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:849)
>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:778)
>     org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 207.69 s <<<
FAILURE! - in org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest  Time elapsed:
207.69 s  <<< ERROR!
> java.io.IOException: Shutting down
> 	at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> Caused by: java.lang.RuntimeException: Master not initialized after 200000ms seconds
> 	at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> 
> 
> Diffs
> -----
> 
>   hbase-agent/src/test/java/org/apache/ranger/authorization/hbase/HBaseRangerAuthorizationTest.java
172b310f1 
>   hive-agent/src/test/java/org/apache/ranger/services/hive/HIVERangerAuthorizerTest.java
62381743f 
>   pom.xml 40b053bd2 
>   security-admin/pom.xml 76bb317e5 
>   src/main/assembly/admin-web.xml c862155f3 
> 
> 
> Diff: https://reviews.apache.org/r/70677/diff/3/
> 
> 
> Testing
> -------
> 
> Tested Ranger Build, Ranger installation and Ranger admin service, zone, policy, user
and group crud operation.
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message