hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshat Mahajan <amaha...@brightedge.com>
Subject Hbase REST returns invalid status code
Date Wed, 08 Feb 2017 02:29:17 GMT
We are seeing exceptions where Hbase REST is returning an invalid status code on writes, causing
our Python processes to error out on trying to make a write (the following error is thrown
by the Python starbase library, but is not caused by starbase directly):

`requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))`

There are no traces nor errors reported in the REST log at the time of the exception. We know
that this is consistent, as we retry thrice with exponential backoff (2 seconds initially).

Our REST threads are allowed to grow dynamically from 100 to 9000 at worst; our connections
typically do not exceed these values, and did not at the time.

Are there circumstances where Hbase REST will deliberately refuse to honor a connection or
respond with invalid codes? Or should I continue looking for more environmental reasons? Note
that at the time REST was documenting wait on a number of AsyncProcesses on several tasks
after having recently closed a table - not sure how useful that information is. A log snippet
at the time from Hbase REST is below:

```
2017-02-07 17:07:35,312 INFO  [196541187@qtp-1054818236-2084] client.HBaseAdmin: Started disable
of urls_table_50007_20170207170255
2017-02-07 17:07:43,294 INFO  [196541187@qtp-1054818236-2084] client.HBaseAdmin: Disabled
urls_table_50007_20170207170255
2017-02-07 17:07:44,093 INFO  [196541187@qtp-1054818236-2084] client.HBaseAdmin: Deleted urls_table_50007_20170207170255
2017-02-07 17:07:53,238 INFO  [115866407@qtp-1054818236-2339] client.HBaseAdmin: Started disable
of urls_table_50001_20170207170502
2017-02-07 17:07:59,953 INFO  [115866407@qtp-1054818236-2339] client.HBaseAdmin: Disabled
urls_table_50001_20170207170502
2017-02-07 17:08:02,128 INFO  [115866407@qtp-1054818236-2339] client.HBaseAdmin: Deleted urls_table_50001_20170207170502
2017-02-07 17:08:04,585 INFO  [328489809@qtp-1054818236-2207] client.HBaseAdmin: Started disable
of urls_table_50023_20170207170455
2017-02-07 17:08:04,791 INFO  [1227614359@qtp-1054818236-647] client.HBaseAdmin: Started disable
of urls_table_50028_20170207170311
2017-02-07 17:08:05,664 INFO  [1252180146@qtp-1054818236-143] client.HBaseAdmin: Started disable
of urls_table_50036_20170207170458
2017-02-07 17:08:07,783 INFO  [328489809@qtp-1054818236-2207] client.HBaseAdmin: Disabled
urls_table_50023_20170207170455
2017-02-07 17:08:14,066 INFO  [1474938506@qtp-1054818236-2246] client.AsyncProcess: #1361530,
waiting for some tasks to finish. Expected max=0, tasksInProgress=7
2017-02-07 17:08:14,167 INFO  [865578926@qtp-1054818236-1712] client.AsyncProcess: #1361562,
waiting for some tasks to finish. Expected max=0, tasksInProgress=7
2017-02-07 17:08:14,267 INFO  [1276865244@qtp-1054818236-1407] client.AsyncProcess: #1361556,
waiting for some tasks to finish. Expected max=0, tasksInProgress=7
2017-02-07 17:08:14,267 INFO  [1368235409@qtp-1054818236-315] client.AsyncProcess: #1361671,
waiting for some tasks to finish. Expected max=0, tasksInProgress=7
```

Akshat

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message