lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Date Tue, 18 Jul 2017 13:37:15 GMT
Hello Susheel,

Yes, the closing happens only at the end of the checking cycle. I asked my colleague about
the firewall and he is positive everything is allowed between those nodes. I also cannot completely
drop the firewall between those nodes to be sure, because the problem is very hard to reproduce;
it pops up once in a while, sometimes not for weeks, today already a couple of times.

It is locally unreproducible but we're going to try to reproduce it in our development environment.
So i have to get back to the problem in a few weeks from now.

Number of requests is sometimes, briefly, very high. Usually very low. These specific checks
are executed in order, not concurrently.

Thanks,
Markus
 
-----Original message-----
> From:Susheel Kumar <susheel2777@gmail.com>
> Sent: Tuesday 18th July 2017 15:17
> To: solr-user@lucene.apache.org
> Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
> 
> Then most likely its due to closing of connection as mentioned above though
> you said it's not happening in that part of your code.  To rule out
> firewall possibility, you can test in some other/local env.  Also how many
> requests/client/connections happening concurrently.
> 
> Thanks,
> Susheel
> 
> On Tue, Jul 18, 2017 at 8:43 AM, Markus Jelsma <markus.jelsma@openindex.io>
> wrote:
> 
> > Hello Susheel,
> >
> > No, nothing at all. I've check all six nodes, they are clean.
> >
> > Thanks,
> > Markus
> >
> >
> >
> > -----Original message-----
> > > From:Susheel Kumar <susheel2777@gmail.com>
> > > Sent: Tuesday 18th July 2017 14:30
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
> > >
> > > Do you see any errors etc. in solr.log during this time?
> > >
> > > On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma <
> > markus.jelsma@openindex.io>
> > > wrote:
> > >
> > > > The problem was never resolved but Shawn asked for the stack trace,
> > here
> > > > it is:
> > > >
> > > > org.apache.solr.client.solrj.SolrServerException: java.lang.
> > IllegalStateException:
> > > > Connection pool shut down
> > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > doRequest(LBHttpSolrClient.java:485)
> > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > > > LBHttpSolrClient.java:388)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.
> > > > sendRequest(CloudSolrClient.java:1383)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.
> > > > requestWithRetryOnStaleState(CloudSolrClient.java:1134)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> > > > CloudSolrClient.java:1073)
> > > > at org.apache.solr.client.solrj.SolrRequest.process(
> > SolrRequest.java:160)
> > > > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
> > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.
> > java:1173)
> > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.
> > java:1090)
> > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.
> > java:1110)
> > > >         ..internal method calling getById()..
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > Caused by: java.lang.IllegalStateException: Connection pool shut down
> > > > at org.apache.http.util.Asserts.check(Asserts.java:34)
> > > > at org.apache.http.pool.AbstractConnPool.lease(
> > AbstractConnPool.java:184)
> > > > at org.apache.http.pool.AbstractConnPool.lease(
> > AbstractConnPool.java:217)
> > > > at org.apache.http.impl.conn.PoolingClientConnectionManager
> > > > .requestConnection(PoolingClientConnectionManager.java:184)
> > > > at org.apache.http.impl.client.DefaultRequestDirector.execute(
> > > > DefaultRequestDirector.java:415)
> > > > at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> > > > AbstractHttpClient.java:882)
> > > > at org.apache.http.impl.client.CloseableHttpClient.execute(
> > > > CloseableHttpClient.java:82)
> > > > at org.apache.http.impl.client.CloseableHttpClient.execute(
> > > > CloseableHttpClient.java:55)
> > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > executeMethod(HttpSolrClient.java:515)
> > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > HttpSolrClient.java:279)
> > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > HttpSolrClient.java:268)
> > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > doRequest(LBHttpSolrClient.java:447)
> > > > ... 24 more
> > > >
> > > > So, to summarize, we have a program checking presence of documents in
> > Solr
> > > > using getById() and we don't want this exception to bubble up, we want
> > > > SolrJ to restore the connection pool just as CloudSolrClient would
> > move on
> > > > to another node if one went down in the mean time.
> > > >
> > > > Is this possible? How?
> > > >
> > > > Many thanks,
> > > > Markus
> > > >
> > > > -----Original message-----
> > > > > From:Markus Jelsma <markus.jelsma@openindex.io>
> > > > > Sent: Thursday 29th June 2017 16:38
> > > > > To: solr-user@lucene.apache.org
> > > > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown
> > > > >
> > > > > Thanks. I probably should have mentioned there is no firewall
> > limiting
> > > > connections between those hosts. Actually, the processes run on the
> > same
> > > > hosts as the Solr cluster is running on.
> > > > >
> > > > > Thanks,
> > > > > Markus
> > > > >
> > > > >
> > > > >
> > > > > -----Original message-----
> > > > > > From:Alexandre Rafalovitch <arafalov@gmail.com>
> > > > > > Sent: Thursday 29th June 2017 15:38
> > > > > > To: solr-user <solr-user@lucene.apache.org>
> > > > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > > > > >
> > > > > > One thing to check is whether there is a firewall between the
> > client
> > > > > > and the server. They - sometimes - cut the silent connections
in
> > the
> > > > > > _middle_ (at the firewall). The usual solution is keepAlive
> > request of
> > > > > > some kind or not using the connection pool.
> > > > > >
> > > > > > One way to check is with network tracer like Wireshark and checking
> > > > > > whether the actual hardware at the other end of the connection
is a
> > > > > > normal server or some sort of unexpected hardware piece of
> > equipment
> > > > > > (firewall). Yes, that's using the hammer to swat a fly :-)
> > > > > >
> > > > > > Regards,
> > > > > >    Alex.
> > > > > > ----
> > > > > > http://www.solr-start.com/ - Resources for Solr users, new and
> > > > experienced
> > > > > >
> > > > > >
> > > > > > On 29 June 2017 at 08:21, Markus Jelsma <
> > markus.jelsma@openindex.io>
> > > > wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Everything is 6.6.0. I could include a stack trace (i don't
print
> > > > them in my program), but that would only be the the trace from
> > getById() to
> > > > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that
> > > > what you're looking for?
> > > > > > >
> > > > > > > We haven't called close() in that particular part of the
program.
> > > > > > >
> > > > > > > Method requestWithRetryOnStaleState has some retry logic
built-in
> > > > but doesn't seem to work for the exception i got.
> > > > > > >
> > > > > > > I'll let it print the stack trace and get back if it happens
> > again.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Markus
> > > > > > >
> > > > > > > -----Original message-----
> > > > > > >> From:Shawn Heisey <apache@elyograg.org>
> > > > > > >> Sent: Tuesday 27th June 2017 23:02
> > > > > > >> To: solr-user@lucene.apache.org
> > > > > > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > > > > > >>
> > > > > > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote:
> > > > > > >> > We have a proces checking presence of many documents
in a
> > > > collection, just a simple client.getById(id). It sometimes begins
> > throwing
> > > > lots of  these exceptions in a row:
> > > > > > >> >
> > > > > > >> > org.apache.solr.client.solrj.SolrServerException:
java.lang.
> > IllegalStateException:
> > > > Connection pool shut down
> > > > > > >> >
> > > > > > >> > Then, as suddenly as it appeared, it's gone again
a no longer
> > a
> > > > problem. I would expect SolrJ not to throw this but to wait until it
> > the
> > > > connection pool, or whatever mechanism is there, to recover.
> > > > > > >> >
> > > > > > >> > Did i miss a magic parameter for SolrJ?\
> > > > > > >>
> > > > > > >> That error message will be much longer than what you've
provided
> > > > here.
> > > > > > >> It will have a java stacktrace that's typically a dozen
or so
> > lines
> > > > > > >> long.  There may also be one or more "Caused by" sections
after
> > the
> > > > > > >> stacktrace, each with a stacktrace of its own.  Can
you share
> > the
> > > > full
> > > > > > >> error message?  Is the server also running 6.6.0, or
a different
> > > > version?
> > > > > > >>
> > > > > > >> It would also be helpful if you can share the SolrJ
code you've
> > > > written,
> > > > > > >> cleanly redacted to remove anything sensitive.
> > > > > > >>
> > > > > > >> That particular message ("Connection pool shut down")
sounds
> > like it
> > > > > > >> probably came from HttpClient, which SolrJ uses ...
and I would
> > > > expect
> > > > > > >> that to only happen if you close/shutdown the HttpClient
or the
> > > > > > >> SolrClient.  After closing, a client can't be used
any more.
> > > > Normally
> > > > > > >> the only time you should close a client is right before
exiting
> > the
> > > > > > >> program, although if the program's about to exit, it's
generally
> > > > > > >> unnecessary, so in my opinion for *most* usages, closing
the
> > client
> > > > > > >> likely never needs to happen.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Shawn
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Mime
View raw message