lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Woodward <a...@flax.co.uk>
Subject Re: SolrCloud: A previous ephemeral live node still exists
Date Wed, 31 Aug 2016 08:43:51 GMT
It looks as though all four nodes are trying to register with ZK using the same hostname and
port number - possibly they're all connecting as 'localhost'?

Alan Woodward
www.flax.co.uk


On 31 Aug 2016, at 09:34, Chris Rogers wrote:

> Just pinging this again as I sent it late last night. Would be great if someone could
help with this. It's got me totally stumped...
> 
> 
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rogers@bodleian.ox.ac.uk
> 
> ________________________________________
> From: Chris Rogers [chris.rogers@bodleian.ox.ac.uk]
> Sent: 30 August 2016 21:34
> To: solr-user@lucene.apache.org
> Subject: SolrCloud: A previous ephemeral live node still exists
> 
> Hi all,
> 
> I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper
3.4.8
> 
> I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more
than one node in cloud mode with Solr 6.2.0.
> 
> I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper,
and three Solr nodes.
> 
> The first Solr node connects to Zookeeper as expected. I can access the admin and see
the Cloud info there.
> 
> But when I try and connect with a second node, something goes wrong.
> 
> I get the following error in the Solr log:
> 
> 
> 2016-08-30 20:12:12.292 INFO  (main) [   ] o.e.j.u.log Logging initialized @490ms
> 
> 2016-08-30 20:12:12.551 INFO  (main) [   ] o.e.j.s.Server jetty-9.3.8.v20160314
> 
> 2016-08-30 20:12:12.574 INFO  (main) [   ] o.e.j.d.p.ScanningAppProvider Deployment monitor
[file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0
> 
> 2016-08-30 20:12:12.906 INFO  (main) [   ] o.e.j.w.StandardDescriptorProcessor NO JSP
Support for /solr, did not find org.apache.jasper.servlet.JspServlet
> 
> 2016-08-30 20:12:12.923 WARN  (main) [   ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
has uncovered http methods for path: /
> 
> 2016-08-30 20:12:12.936 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init():
WebAppClassLoader=1465085305@57536d79
> 
> 2016-08-30 20:12:12.962 INFO  (main) [   ] o.a.s.c.SolrResourceLoader JNDI not configured
for solr (NoInitialContextEx)
> 
> 2016-08-30 20:12:12.962 INFO  (main) [   ] o.a.s.c.SolrResourceLoader using system property
solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
> 
> 2016-08-30 20:12:12.963 INFO  (main) [   ] o.a.s.c.SolrResourceLoader new SolrResourceLoader
for directory: '/home/vagrant/solr-6.2.0/server/solr'
> 
> 2016-08-30 20:12:12.963 INFO  (main) [   ] o.a.s.c.SolrResourceLoader JNDI not configured
for solr (NoInitialContextEx)
> 
> 2016-08-30 20:12:12.963 INFO  (main) [   ] o.a.s.c.SolrResourceLoader using system property
solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
> 
> 2016-08-30 20:12:12.987 INFO  (main) [   ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
> 
> 2016-08-30 20:12:13.017 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Waiting for client
to connect to ZooKeeper
> 
> 2016-08-30 20:12:13.118 INFO  (zkCallback-1-thread-1) [   ] o.a.s.c.c.ConnectionManager
Watcher org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181
got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> 
> 2016-08-30 20:12:13.119 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Client is connected
to ZooKeeper
> 
> 2016-08-30 20:12:13.119 INFO  (main) [   ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
> 
> 2016-08-30 20:12:13.130 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Loading solr.xml
from SolrHome (not found in ZooKeeper)
> 
> 2016-08-30 20:12:13.133 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading container configuration
from /home/vagrant/solr-6.2.0/server/solr/solr.xml
> 
> 2016-08-30 20:12:13.233 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Config-defined
core root directory: /home/vagrant/solr-6.2.0/server/solr
> 
> 2016-08-30 20:12:13.264 INFO  (main) [   ] o.a.s.c.CoreContainer New CoreContainer 182259421
> 
> 2016-08-30 20:12:13.265 INFO  (main) [   ] o.a.s.c.CoreContainer Loading cores into CoreContainer
[instanceDir=/home/vagrant/solr-6.2.0/server/solr]
> 
> 2016-08-30 20:12:13.266 WARN  (main) [   ] o.a.s.c.CoreContainer Couldn't add files from
/home/vagrant/solr-6.2.0/server/solr/lib to classpath: /home/vagrant/solr-6.2.0/server/solr/lib
> 
> 2016-08-30 20:12:13.281 INFO  (main) [   ] o.a.s.h.c.HttpShardHandlerFactory created
with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections
: 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue :
-1,fairnessPolicy : false,useRetries : false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime
: 40000,
> 
> 2016-08-30 20:12:13.507 INFO  (main) [   ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler
HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
> 
> 2016-08-30 20:12:13.511 INFO  (main) [   ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
> 
> 2016-08-30 20:12:13.512 INFO  (main) [   ] o.a.s.l.LogWatcher Registering Log Listener
[Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> 
> 2016-08-30 20:12:13.519 INFO  (main) [   ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181
> 
> 2016-08-30 20:12:13.535 INFO  (main) [   ] o.a.s.c.ZkController Added new OnReconnect
listener org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
> 
> 2016-08-30 20:12:13.538 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Waiting for client
to connect to ZooKeeper
> 
> 2016-08-30 20:12:13.549 INFO  (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr)
[   ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
> 
> 2016-08-30 20:12:13.549 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Client is connected
to ZooKeeper
> 
> 2016-08-30 20:12:13.620 INFO  (main) [   ] o.a.s.c.c.ZkStateReader Updating cluster state
from ZooKeeper...
> 
> 2016-08-30 20:12:13.627 INFO  (main) [   ] o.a.s.c.c.ZkStateReader Loaded empty cluster
properties
> 
> 2016-08-30 20:12:13.634 INFO  (main) [   ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (0) -> (1)
> 
> 2016-08-30 20:12:43.674 ERROR (main) [   ] o.a.s.s.SolrDispatchFilter Could not start
Solr. Check solr/home property and the logs
> 
> 2016-08-30 20:12:43.741 ERROR (main) [   ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException:
A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other
Solr process using the same port is running already.
> 
> at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
> 
> at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
> 
> at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
> 
> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
> 
> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
> 
> at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
> 
> at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
> 
> at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
> 
> at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
> 
> at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
> 
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
> 
> at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
> 
> at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
> 
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
> 
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
> 
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> 
> at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
> 
> at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
> 
> at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
> 
> at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
> 
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
> 
> at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
> 
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
> 
> at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
> 
> at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
> 
> at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
> 
> at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
> 
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> 
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
> 
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> 
> at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
> 
> at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
> 
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> 
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> 
> at org.eclipse.jetty.server.Server.start(Server.java:405)
> 
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> 
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> 
> at org.eclipse.jetty.server.Server.doStart(Server.java:372)
> 
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> 
> at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
> 
> at java.security.AccessController.doPrivileged(Native Method)
> 
> at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
> 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 
> at java.lang.reflect.Method.invoke(Method.java:498)
> 
> at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
> 
> at org.eclipse.jetty.start.Main.start(Main.java:457)
> 
> at org.eclipse.jetty.start.Main.main(Main.java:75)
> 
> 
> 2016-08-30 20:12:43.744 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init()
done
> 
> 2016-08-30 20:12:43.762 INFO  (main) [   ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
> 
> 2016-08-30 20:12:43.771 INFO  (main) [   ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
> 
> 2016-08-30 20:12:43.771 INFO  (main) [   ] o.e.j.s.Server Started @31972ms
> 
> It does start in the end, but it's not working properly, I just get errors in the Solr
admin.
> 
> The only reference I can find to the the error:
> 
> "A previous ephemeral live node still exists. Solr cannot continue. Please ensure that
no other Solr process using the same port is running already."
> 
> ,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
> 
> https://issues.apache.org/jira/browse/SOLR-8777
> 
> Has anyone else experienced this issue? Any hints on resolving?
> 
> 
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rogers@bodleian.ox.ac.uk<mailto://chris.rogers@bodleian.ox.ac.uk>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message