hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Hoppins <marc.hopp...@eset.sk>
Subject RE: Region server idle
Date Fri, 08 Jan 2021 10:51:18 GMT
Apologies again.  Here is the full error message.

2021-01-08 11:34:15,831 ERROR org.apache.hadoop.hbase.ScheduledChore: Caught error
java.lang.IllegalStateException: Expected only one of namespace and tablename to be null
	at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:198)
	at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:126)
	at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:113)
	at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

-----Original Message-----
From: Marc Hoppins <marc.hoppins@eset.sk> 
Sent: Friday, January 8, 2021 10:57 AM
To: user@hbase.apache.org
Subject: RE: Region server idle

EXTERNAL

So, I tried decommission that RS and recommission it.  No change. Server still idle.

Tried decommission another server and see if HBASE sets itself right. Now I have two RS that
are idle.

ba-hbase18.jumbo.hq.com,16020,1604413480001     Tue Nov 03 15:24:40 CET 2020    1 s     2.1.0-cdh6.3.2
 13      471
ba-hbase19.jumbo.hq.com,16020,1610095488001     Fri Jan 08 09:44:48 CET 2021    0 s     2.1.0-cdh6.3.2
 0       6
ba-hbase20.jumbo.hq.com,16020,1610096850259     Fri Jan 08 10:07:30 CET 2021    0 s     2.1.0-cdh6.3.2
 0       0
ba-hbase21.jumbo.hq.com,16020,1604414101652     Tue Nov 03 15:35:01 CET 2020    1 s     2.1.0-cdh6.3.2
 15      447

From the logs:
2021-01-08 10:25:36,875 ERROR org.apache.hadoop.hbase.ScheduledChore: Caught error java.lang.IllegalStateException:
Expected only one of namespace and tablename to be null

This is reappearing in hbase master log

M

-----Original Message-----
From: Sean Busbey <busbey@apache.org>
Sent: Thursday, January 7, 2021 7:30 PM
To: Hbase-User <user@hbase.apache.org>
Subject: Re: Region server idle

EXTERNAL

Sounds like https://issues.apache.org/jira/browse/HBASE-24139

The description of that jira has a workaround.

On Thu, Jan 7, 2021, 05:23 Marc Hoppins <marc.hoppins@eset.sk> wrote:

> Hi all,
>
> I have a setup with 67 region servers. 29 Dec one system had to be 
> shut down to have EMM module swapped out - which took one work day.
> Host was back online 30 Dec.
>
> My HBASE is very basic so I appreciate your patience.
>
> My understanding of the defaults that are setup is that a major 
> compaction should occur every 7 days.  Moreover, do I assume that more 
> extensive balancing may occur after this happens?
>
> When I check (via hbase master UI) the status of HBASE, I see the
> following:
>
> ServerName
>
> Start time
>
> Last contact
>
> Version
>
> Requests Per Second
>
> Num. Regions
>
> ba-hbase16.jumbo.hq.com,16020,1604413068640<
> http://ba-hbase16.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:17:48 CET 2020
>
> 3 s
>
> 2.1.0-cdh6.3.2
>
> 46
>
> 462
>
> ba-hbase17.jumbo.hq.com,16020,1604413274393<
> http://ba-hbase17.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:21:14 CET 2020
>
> 1 s
>
> 2.1.0-cdh6.3.2
>
> 19
>
> 462
>
> ba-hbase18.jumbo.hq.com,16020,1604413480001<
> http://ba-hbase18.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:24:40 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 62
>
> 461
>
> ba-hbase19.jumbo.hq.com,16020,1609326754985<
> http://ba-hbase19.jumbo.hq.eset.com:16030/rs-status>
>
> Wed Dec 30 12:12:34 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 0
>
> 0
>
> ba-hbase20.jumbo.hq.com,16020,1604413895967<
> http://ba-hbase20.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:31:35 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 62
>
> 503
>
> ba-hbase21.jumbo.hq.com,16020,1604414101652<
> http://ba-hbase21.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:35:01 CET 2020
>
> 3 s
>
> 2.1.0-cdh6.3.2
>
> 59
>
> 442
>
> ba-hbase22.jumbo.hq.com,16020,1604414308289<
> http://ba-hbase22.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:38:28 CET 2020
>
> 0 s
>
> 2.1.0-cdh6.3.2
>
> 40
>
> 438
>
>
> Why, after more than 7 days, is this host not hosting more (any) regions?
> Should I initiate some kind of rebalancing?
>
> Thanks in advance.
>
> M
>
Mime
View raw message