trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Varnau <steve.var...@esgyn.com>
Subject RE: Trafodion master Daily Test Result - 211 - Failure
Date Thu, 19 May 2016 23:54:19 GMT
In the test job log[1], at the beginning, there were 7GB free on the
filesystem, but that is before installing trafodion and running the executor
suite.

At the end of the job, after completing the suite and uninstalling
trafodion, there were 6 GB free.

Hard to say how much was free when that test ran, but not hard to imagine
that it was less than 2GB.

At beginning, it also lists disk usage of the usual suspects (HDFS data and
logs):
2016-05-19 10:23:38 Disk usage of specific directories
2016-05-19 10:23:38 Thu May 19 10:23:38 UTC 2016:  1.4G	/hadoop/hdfs
2016-05-19 10:23:38 Thu May 19 10:23:38 UTC 2016:  29G	/var/log

This machine may have run several test suites, but normally the same every
night. So how much space was available on the previous day's job?

38GB were free, and /var/log was using only 6.8MB, not 29GB.

So that says to me, even though tests were generally passing, something was
spewing log messages!
Looking at the archived HBase logs, they are nice and small.

I'll bring up a VM with the disk volume in question to see where all the
space went.

--Steve

[1] https://jenkins.esgyn.com/job/core-regress-executor-hdp/220/console

> -----Original Message-----
> From: Prashanth Vasudev [mailto:prashanth.vasudev@esgyn.com]
> Sent: Thursday, May 19, 2016 4:01 PM
> To: dev@trafodion.incubator.apache.org; no-reply@trafodion.org
> Subject: RE: Trafodion master Daily Test Result - 211 - Failure
>
> System error 28 is file system error returned when Disk space ( where
> overflow disk is configured) is less than 2 gb available.
> By default overflow location is $My_SQROOT/tmp. The disk that is hosting
> this file system is getting out of disk space.
>
> Prashanth
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Thursday, May 19, 2016 12:09 PM
> To: dev@trafodion.incubator.apache.org; no-reply@trafodion.org
> Subject: RE: Trafodion master Daily Test Result - 211 - Failure
>
> It looks like executor/TEST002 failed due to no space in swap.
>
> Error shown in the diff is
>
> > *** ERROR[8427] Hash Join Scratch IO Error occurred. Scratch
> > Error: -10005, System Error: 28, System Error Detail: 0, Details:
> > SQScratchFile::SQScratchFile(5)
> 4178c3135
>
> System Error 28 is     #define ENOSPC 28
>
> By default sort overflow mode is MMAP. So, I suspect the node is running
> out
> of swap space at that moment.
>
> Selva
> -----Original Message-----
> From: steve.varnau@esgyn.com [mailto:steve.varnau@esgyn.com]
> Sent: Thursday, May 19, 2016 4:47 AM
> To: dev@trafodion.incubator.apache.org
> Subject: Trafodion master Daily Test Result - 211 - Failure
>
> Daily Automated Testing master
>
> Jenkins Job:   https://jenkins.esgyn.com/job/Check-Daily-master/211/
> Archived Logs: http://traf-testlogs.esgyn.com/Daily-master/211
> Bld Downloads: http://traf-builds.esgyn.com
>
> Changes since previous daily build:
> [selvaganesang] [TRAFODION-1988] Better Java exception handling in
> Trafodion
>
> [weiqing.xu] [TRAFODION-1998]JDBC Type2 column information improve
>
>
>
> Test Job Results:
>
> FAILURE core-regress-executor-hdp (1 hr 23 min) SUCCESS build-master-debug
> (26 min) SUCCESS build-master-release (30 min) SUCCESS
> core-regress-charsets-cdh (29 min) SUCCESS core-regress-charsets-hdp (37
> min) SUCCESS core-regress-compGeneral-cdh (32 min) SUCCESS
> core-regress-compGeneral-hdp (48 min) SUCCESS core-regress-core-cdh (47
> min)
> SUCCESS core-regress-core-hdp (1 hr 11 min) SUCCESS
> core-regress-executor-cdh (52 min) SUCCESS core-regress-fullstack2-cdh (10
> min) SUCCESS core-regress-fullstack2-hdp (26 min) SUCCESS
> core-regress-hive-cdh (36 min) SUCCESS core-regress-hive-hdp (51 min)
> SUCCESS core-regress-privs1-cdh (36 min) SUCCESS core-regress-privs1-hdp
> (57
> min) SUCCESS core-regress-privs2-cdh (41 min) SUCCESS
> core-regress-privs2-hdp (54 min) SUCCESS core-regress-qat-cdh (20 min)
> SUCCESS core-regress-qat-hdp (28 min) SUCCESS core-regress-seabase-cdh (53
> min) SUCCESS core-regress-seabase-hdp (1 hr 13 min) SUCCESS
> core-regress-udr-cdh (25 min) SUCCESS core-regress-udr-hdp (31 min)
> SUCCESS
> jdbc_test-cdh (28 min) SUCCESS jdbc_test-hdp (32 min) SUCCESS
> phoenix_part1_T2-cdh (1 hr 1 min) SUCCESS phoenix_part1_T2-hdp (1 hr 18
> min)
> SUCCESS phoenix_part1_T4-cdh (46 min) SUCCESS phoenix_part1_T4-hdp (58
> min)
> SUCCESS phoenix_part2_T2-cdh (51 min) SUCCESS phoenix_part2_T2-hdp (1 hr
> 28
> min) SUCCESS phoenix_part2_T4-cdh (46 min) SUCCESS phoenix_part2_T4-hdp
> (1
> hr 7 min) SUCCESS pyodbc_test-cdh (16 min) SUCCESS pyodbc_test-hdp (14
> min)

Mime
View raw message