lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar <susheel2...@gmail.com>
Subject Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Date Wed, 04 May 2016 22:13:43 GMT
Thanks, Nick. Do we know any suggested # for file descriptor limit with
Solr6?  Also wondering why i haven't seen this problem before with Solr 5.x?

On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <nick.vasilyev1@gmail.com>
wrote:

> It looks like you have too many open files, try increasing the file
> descriptor limit.
>
> On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <susheel2777@gmail.com>
> wrote:
>
> > Hello,
> >
> > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and used
> the
> > install service to setup solr.
> >
> > After launching Solr Admin Panel on server1, it looses connections in few
> > seconds and then comes back and other node server2 is marked as Down in
> > cloud graph. After few seconds its loosing the connection and comes back.
> >
> > Any idea what may be going wrong? Has anyone used Solr 6 with ZK 3.4.8.
> > Have never seen this error before with solr 5.x with ZK 3.4.6.
> >
> > Below log from server1 & server2.  The ZK has 3 nodes with chroot
> enabled.
> >
> > Thanks,
> > Susheel
> >
> > server1/solr.log
> >
> > ====
> >
> >
> > 2016-05-04 19:20:53.804 INFO  (qtp1989972246-14) [   ]
> > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
> > [configName]=[collection1] specified config exists in ZooKeeper
> >
> > 2016-05-04 19:20:53.806 INFO  (qtp1989972246-14) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
> >
> > 2016-05-04 19:20:53.859 INFO  (qtp1989972246-19) [   ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:53.861 INFO  (qtp1989972246-19) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> >
> > 2016-05-04 19:20:57.520 INFO  (qtp1989972246-13) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> >
> > 2016-05-04 19:20:57.546 INFO  (qtp1989972246-15) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389588126} status=0 QTime=25
> >
> > 2016-05-04 19:20:57.610 INFO  (qtp1989972246-13) [   ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:57.613 INFO  (qtp1989972246-13) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> >
> > 2016-05-04 19:21:29.139 INFO  (qtp1989972246-5980) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.141 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.141 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.142 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO  (qtp1989972246-5980) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.143 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.144 INFO  (qtp1989972246-5980) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:20:53.806 INFO  (qtp1989972246-14) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
> >
> > 2016-05-04 19:20:53.859 INFO  (qtp1989972246-19) [   ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:53.861 INFO  (qtp1989972246-19) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> >
> > 2016-05-04 19:20:57.520 INFO  (qtp1989972246-13) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> >
> > 2016-05-04 19:20:57.546 INFO  (qtp1989972246-15) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389588126} status=0 QTime=25
> >
> > 2016-05-04 19:20:57.610 INFO  (qtp1989972246-13) [   ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:57.613 INFO  (qtp1989972246-13) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> >
> > 2016-05-04 19:21:29.139 INFO  (qtp1989972246-5980) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.141 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.141 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.142 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO  (qtp1989972246-5984) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO  (qtp1989972246-5980) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.143 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.144 INFO  (qtp1989972246-5980) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.145 INFO  (qtp1989972246-5983) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.146 INFO  (qtp1989972246-5985) [   ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > …
> >
> > …
> >
> > 2016-05-04 19:21:29.150 INFO  (qtp1989972246-5980) [   ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [   ]
> > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> > trying to proxy request for url:
> > http://server2:8983/solr/collection1/admin/luke
> >
> >         at
> > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
> >
> >         at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
> >
> >         at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> >
> >         at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> >
> >         at
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> >
> >         at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> >
> >         at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >
> >         at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >
> >         at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> >
> >         at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> >
> >
> >
> >
> >
> > server2/solr.log
> >
> > ======
> >
> >
> > 016-05-04 18:14:32.936 INFO  (main) [   ] o.e.j.s.ServerConnector Started
> > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
> >
> > 2016-05-04 18:14:32.936 INFO  (main) [   ] o.e.j.s.Server Started @3850ms
> >
> > 2016-05-04 19:20:20.243 INFO  (qtp1989972246-18) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
> >
> > 2016-05-04 19:20:20.254 INFO  (qtp1989972246-15) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/zookeeper
> > params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
> >
> > 2016-05-04 19:20:20.264 INFO  (qtp1989972246-20) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389620002} status=0 QTime=50
> >
> > 2016-05-04 19:20:20.321 INFO  (qtp1989972246-19) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/zookeeper
> >
> >
> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
> > status=0 QTime=6
> >
> > 2016-05-04 19:20:20.360 INFO  (qtp1989972246-21) [   ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:20.362 INFO  (qtp1989972246-21) [   ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
> >
> > 2016-05-04 19:34:35.760 WARN
> > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
> > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [   ]
> > o.e.j.s.ServerConnector
> >
> > java.io.IOException: Too many open files
> >
> >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> >
> >         at
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> >
> >         at
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> >
> >         at
> > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
> >
> >         at
> >
> >
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
> >
> >         at
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> >
> >         at
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >
> >         at java.lang.Thread.run(Thread.java:745)
> >
> > 2016-05-04 19:38:09.442 INFO  (qtp1989972246-5289) [   ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
> >
> > 2016-05-04 19:38:09.496 INFO  (qtp1989972246-1224) [   ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
> >
> > 2016-05-04 19:38:09.587 INFO  (qtp1989972246-5765) [   ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389620002} status=0 QTime=99
> >
> > 2016-05-04 19:38:09.656 INFO  (qtp1989972246-4352) [   ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> >
> > 2016-05-04 19:38:09.658 INFO  (qtp1989972246-4352) [   ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
> >
> > 2016-05-04 19:38:19.442 INFO  (qtp1989972246-4324) [   ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message