lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9828 - Failure!
Date Tue, 25 Mar 2014 08:14:20 GMT
Hi Mark,

 

last night this happened again (for the 3rd time). For the second time in a windows VM. Maybe
it was another test than the one we have seen it first.

 

I don’t think disabling SSL helps here. It happens together with SSL, that’s right, but
from the log file there seems to be some bug in the test setup: The test tries to reconnect
endless without stopping to try again and again and later failing the test. This fills disk
space quite fast. This also makes the tests never end.

 

The rate of reconnects is so high, that the log file is filled with megabytes in very short
time. When disk is full, the carrot framework is no longer able to handle this case and the
whole JVM setup hangs.

 

Unfortunately, I have no data available anymore because I had to revert the Windows Virtualbox
VM to the latest clean snapshot:

 

[junit4] Could not serialize report for suite org.apache.solr.cloud.TestShortCircuitedRequests:
java.io.IOException: There is not enough space on the disk

   [junit4] Mar 24, 2014 8:44:09 PM com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler
handleException

   [junit4] SEVERE: Could not dispatch event: com.carrotsearch.ant.tasks.junit4.listeners.TextReport@1290a22
to public void com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent)

 

I am not even sure if this is the test that caused this.

 

Would it be possible to catch those cases while running tests (maybe before the disk is full)
and fail the build? Maybe something that the event file is not allowed to grow beyond a specific
size. If it grows, the test framework fails the whole build? We can have something like maximum
size of 1 GB (configureable).

 

Uwe

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

 <http://www.thetaphi.de/> http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: Mark Miller [mailto:markrmiller@gmail.com] 
Sent: Wednesday, March 19, 2014 6:51 PM
To: dev@lucene.apache.org
Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9828 - Failure!

 

I'll disable SSL for that test for now. SSL in general has been hard to get working smoothly
with tests unfortunately.

 

I've got a JIRA issue to look at improving it, but not likely I'll look into it for some time,
so until then, tests having issues with SSL should likely simply disable SSL for now.

 

- Mark

 

On Tue, Mar 18, 2014 at 4:54 AM, Dawid Weiss <dawid.weiss@cs.put.poznan.pl> wrote:

It's a lot of error messages like this one. I have the full syserr
dump if needed.

D.

2773140 T6223 oasc.ChaosMonkeyNothingIsSafeTest$FullThrottleStopableIndexingThread$1.handleError
WARN suss error java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


On Tue, Mar 18, 2014 at 9:46 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
> I dig "tail -10000" to extract the last 10000 lines. The file is also in the archive
at same place.
>
> It is indeed a loop. The code loops endless in a "Connection Refused" loop, without any
delay between the events. After approx. 2:50 hours this hit the limits of the SSD file system.
This test fails so often since it was "fixed", we should revert to @BadApple.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf
>> Of Dawid Weiss
>> Sent: Tuesday, March 18, 2014 9:16 AM
>> To: dev@lucene.apache.org
>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build #
>> 9828 - Failure!
>>
>> >       junit4-J0-20140317_230107_233.events    8.17 GB [fingerprint] view
>> >
>> > This build created a 8.17 GB big events file and failed with out of space.
>> How can this happen?
>>
>> Can you peek at it? It's probably something that logs in a loop or something.
>> I'm fetching it right now, let's see if I can figure it out.
>>
>> D.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>> commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org





 

-- 

- Mark

 

http://about.me/markrmiller


Mime
View raw message