trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandhya Sundaresan <sandhya.sundare...@esgyn.com>
Subject RE: Trafodion release2.0 Daily Test Result - 23 - Still Failing
Date Thu, 26 May 2016 16:04:19 GMT
 RE: Trafodion release2.0 Daily Test Result - 23 - Still Failing

Hi Steve,

   The error today is this :

 *** ERROR[8448] Unable to access Hbase interface. Call to
ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-704). Cause:

> java.lang.Exception: Cannot create Table Snapshot Scanner

> org.TRAFODION.sql.HTableClient.startScan(HTableClient.java:1003)

We have seen this when there is  java memory pressure in the past.

A few days back this same snapshot scan creation failed with this : I
wonder if anyone can see  pattern here or knows the causes of either of
these.

>>--snapshot

>>execute snp;

*** ERROR[8448] Unable to access Hbase interface. Call to
ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-704). Cause:

java.io.IOException: java.util.concurrent.ExecutionException:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/bulkload/20160520102824/TRAFODION.HBASE.CUSTOMER_ADDRESS_SNAP111/6695c6f9-4bb5-4ad5-893b-adf07fc8a4b9/data/default/TRAFODION.HBASE.CUSTOMER_ADDRESS/7143c21b40a7bef21768685f7dc18e1c/.regioninfo
could only be replicated to 0 nodes instead of minReplication (=1).  There
are 1 datanode(s) running and no node(s) are excluded in this operation.

        at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1541)

        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3289)

        at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:668)

        at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)

        at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)

        at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegions(ModifyRegionUtils.java:162)

org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.cloneHdfsRegions(RestoreSnapshotHelper.java:561)

org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:237)

org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:159)

org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.copySnapshotForScanner(RestoreSnapshotHelper.java:812)

org.apache.hadoop.hbase.client.TableSnapshotScanner.init(TableSnapshotScanner.java:156)

org.apache.hadoop.hbase.client.TableSnapshotScanner.<init>(TableSnapshotScanner.java:124)

org.apache.hadoop.hbase.client.TableSnapshotScanner.<init>(TableSnapshotScanner.java:101)

org.trafodion.sql.HTableClient$SnapshotScanHelper.createTableSnapshotScanner(HTableClient.java:222)

org.trafodion.sql.HTableClient.startScan(HTableClient.java:1009)

.

--- 0 row(s) selected.

>>log;

Sandhya

-----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com <steve.varnau@esgyn.com>]
Sent: Thursday, May 26, 2016 8:49 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: Trafodion release2.0 Daily Test Result - 23 - Still Failing

This hive regression behavior is still puzzling, however, I just realized
one thing that did change just before it started failing and is a test
environment change common to both branches.  The VM image for cloudera was
updated to set a smaller ephemeral port range to reduce chance of port
conflict that was occasionally impacting HBase.

The range was set to 51000 - 59999, to avoid default port numbers that
Cloudera distro uses.

So how could this possibly be causing disaster in hive/TEST018?   I have no

idea.

--Steve

> -----Original Message-----

> From: steve.varnau@esgyn.com [mailto:steve.varnau@esgyn.com
<steve.varnau@esgyn.com>]

> Sent: Thursday, May 26, 2016 1:36 AM

> To: dev@trafodion.incubator.apache.org

> Subject: Trafodion release2.0 Daily Test Result - 23 - Still Failing

>

> Daily Automated Testing release2.0

>

> Jenkins Job:   https://jenkins.esgyn.com/job/Check-Daily-release2.0/23/

> Archived Logs: http://traf-testlogs.esgyn.com/Daily-release2.0/23

> Bld Downloads: http://traf-builds.esgyn.com

>

> Changes since previous daily build:

> No changes

>

>

> Test Job Results:

>

> FAILURE core-regress-hive-cdh (55 min) SUCCESS build-release2.0-debug

> (24 min) SUCCESS build-release2.0-release (28 min) SUCCESS

> core-regress-charsets-cdh (28 min) SUCCESS core-regress-charsets-hdp

> (41 min) SUCCESS core-regress-compGeneral-cdh (36 min) SUCCESS

> core-regress-compGeneral-hdp (45 min) SUCCESS core-regress-core-cdh

> (39 min) SUCCESS core-regress-core-hdp (1 hr 10 min) SUCCESS

> core-regress-executor-cdh (56 min) SUCCESS core-regress-executor-hdp

> (1 hr 25 min) SUCCESS core-regress-fullstack2-cdh (13 min) SUCCESS

> core-regress-fullstack2-hdp (14 min) SUCCESS core-regress-hive-hdp (53

> min) SUCCESS core-regress-privs1-cdh (39 min) SUCCESS

> core-regress-privs1-hdp (59 min) SUCCESS core-regress-privs2-cdh (41

> min) SUCCESS core-regress-privs2-hdp (54 min) SUCCESS

> core-regress-qat-cdh (16 min) SUCCESS core-regress-qat-hdp (21 min)

> SUCCESS core-regress-seabase-cdh (57 min) SUCCESS

> core-regress-seabase-hdp (1 hr 16 min) SUCCESS core-regress-udr-cdh

> (28 min) SUCCESS core-regress-udr-hdp (31 min) SUCCESS jdbc_test-cdh

> (22 min) SUCCESS jdbc_test-hdp (40 min) SUCCESS phoenix_part1_T2-cdh

> (56 min) SUCCESS phoenix_part1_T2-hdp (1 hr 17 min) SUCCESS

> phoenix_part1_T4-cdh (46 min) SUCCESS phoenix_part1_T4-hdp (57 min)

> SUCCESS phoenix_part2_T2-cdh (53 min) SUCCESS phoenix_part2_T2-hdp (1

> hr 25 min) SUCCESS phoenix_part2_T4-cdh (44 min) SUCCESS

> phoenix_part2_T4-hdp (1 hr 0 min) SUCCESS pyodbc_test-cdh (11 min)

> SUCCESS pyodbc_test-hdp (23 min)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message