phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Francis Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PHOENIX-4374) Flakyness with Phoenix 4.13.0 and HBase 1.3.1: RuntimeException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.NamespaceNotFoundException:
Date Wed, 10 Jan 2018 21:40:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321154#comment-16321154
] 

Francis Chuang edited comment on PHOENIX-4374 at 1/10/18 9:39 PM:
------------------------------------------------------------------

Here are the statements from the simplest test that fails. In this test, we run a database
migration to set up the appropriate schema and tables. Once this is done, we drop the tables
and the schema.

{code}
-- Using connection 6b4e63f4-f4f6-472e-a1e6-d9c8c80e5948
CREATE SCHEMA IF NOT EXISTS PHOENIXTEST

-- Using connection 7682303b-7e01-4a69-be4f-44825f28fa0f
SELECT 1
CREATE TABLE IF NOT EXISTS schema_migration (version varchar not null primary key) TRANSACTIONAL=true
SELECT version FROM schema_migration ORDER BY version DESC
CREATE TABLE aggregate(id VARCHAR(255) NOT NULL,bounded_context VARCHAR(255) NOT NULL,aggregate
VARCHAR(255) NOT NULL,organization_id VARCHAR(255) NOT NULL,version BIGINTCONSTRAINT pk PRIMARY
KEY (id, bounded_context, Aggregate, organization_id))TRANSACTIONAL=TRUE
CREATE TABLE event(id BIGINT NOT NULL PRIMARY KEY,created TIMESTAMP,bounded_context VARCHAR(255),aggregate
VARCHAR(255),aggregate_id VARCHAR(255),organization_id VARCHAR(255),event VARCHAR(255),data
VARBINARY,version BIGINT,user_id VARCHAR(255),ip_address VARCHAR(45)) TRANSACTIONAL=TRUE,
SALT_BUCKETS=20
CREATE INDEX ievent ON Event(aggregate_id,bounded_context,aggregate,organization_id,version)
CREATE TABLE snapshot(bounded_context VARCHAR(255) NOT NULL,aggregate VARCHAR(255) NOT NULL,aggregate_id
VARCHAR(255) NOT NULL,organization_id VARCHAR(255) NOT NULL,version BIGINT,created TIMESTAMP,data
VARBINARYCONSTRAINT pk PRIMARY KEY (bounded_context, Aggregate, aggregate_id, organization_id))
CREATE TABLE subscription(id VARCHAR(255) NOT NULL PRIMARY KEY,topic VARCHAR(255),last_event
BIGINT,last_heartbeat TIMESTAMP)
CREATE INDEX topic_subscription ON subscription (topic) INCLUDE (last_event)
CREATE SEQUENCE event_id START WITH 1 INCREMENT BY 1 CACHE 0
UPSERT INTO schema_migration (version) VALUES (?)

-- Using connection 0d08e373-3def-466a-aad7-46180fcbeca7
SELECT 1
CREATE TABLE IF NOT EXISTS schema_migration (version varchar not null primary key) TRANSACTIONAL=true
SELECT version FROM schema_migration ORDER BY version DESC
DROP TABLE IF EXISTS aggregate
DROP TABLE IF EXISTS event
DROP INDEX IF EXISTS ievent ON Event
DROP TABLE IF EXISTS snapshot
DROP TABLE IF EXISTS subscription
DROP INDEX IF EXISTS topic_subscription ON subscription
DROP SEQUENCE IF EXISTS event_id
DELETE FROM schema_migration WHERE version=?

-- Using connection 6b4e63f4-f4f6-472e-a1e6-d9c8c80e5948
DROP TABLE IF EXISTS PHOENIXTEST.schema_migration
DROP SCHEMA IF EXISTS PHOENIXTEST
{code}

I also noticed that this problem tends to surface when there is high load on the machine.


was (Author: francischuang):
Here are the statements from the simplest test that fails. In this test, we run a database
migration to set up the appropriate schema and tables. Once this is done, we drop the tables
and the schema.

{code}

> Flakyness with Phoenix 4.13.0 and HBase 1.3.1:  RuntimeException: org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.NamespaceNotFoundException:
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-4374
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4374
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.13.0
>            Reporter: Francis Chuang
>            Assignee: Thomas D'Silva
>         Attachments: 2017-12-29.m9edd51.log, 2018-01-03.m9edd51.log, 2018-01-05-hbase1.3.m9edd51.log,
hbase--master-m9edd51-phoenix.m9edd51-2017-11-20-second-run.log, hbase--master-m9edd51-phoenix.m9edd51-2017-11-20.log,
hbase--master-m9edd51-phoenix.m9edd51.log, protobufs-dump-2017-11-20-second-run.zip, protobufs-dump-2017-11-20.zip,
root-queryserver-2017-11-20-second-run.log, root-queryserver-2017-11-20.log, root-queryserver.log,
tephra-service--m9edd51-phoenix.m9edd51-2017-11-20-second-run.log, tephra-service--m9edd51-phoenix.m9edd51-2017-11-20.log,
tephra-service--m9edd51-phoenix.m9edd51.log
>
>
> I am using the Phoenix Query Server via my [Go Avatica SQL driver|https://github.com/Boostport/avatica].
> In terms of my set up I am running Phoenix 4.13.0 and HBase 1.3.1 in docker with a single
node HBase using local storage. The dockerfile is available here: https://github.com/Boostport/hbase-phoenix-all-in-one
> Today, I updated one of my projects to use the latest version of the above image (Phoenix
4.13.0 and HBase 1.3.1) and my integration tests against Phoenix + HBase have become extremely
flaky. The tests use a mix of transactional and non-transactional tables.
> The flakyness is that random tests will fail with the same error. If I rerun the tests,
they sometimes pass and sometimes fail, and it is not clear why this is happening.
> In all of these tests, I am doing the following:
> 1. Create the schema.
> 2. Create tables.
> 3. Insert, delete and read data.
> 4. Delete the tables and schema.
> This is the error I get when trying to drop the schema:
> {code:java}
> An error was encountered while processing your request: RuntimeException: org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
>                  -> PhoenixIOException: org.apache.hadoop.hbase.NamespaceNotFoundException:
INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
>                  -> NamespaceNotFoundException: org.apache.hadoop.hbase.NamespaceNotFoundException:
INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
>                  -> RemoteWithExtrasException: org.apache.hadoop.hbase.NamespaceNotFoundException:
INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
> {code}
> I have also attached the following:
> - Logs from tephra
> - Logs from PQS/Avatica
> - Logs from HBase
> Further update:
> Rolled back to the HBase 1.2 Phoenix 4.11 image and tests are rock solid.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message