lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Al Wold <alw...@alwold.com>
Subject Re: Solr cloud: zkHost in solr.xml gets wiped out
Date Tue, 18 Jun 2013 21:47:47 GMT
I just finished a test with the patch, and it looks like all is working well.

On Jun 18, 2013, at 12:19 PM, Al Wold wrote:

> For the CREATE call, I'm doing it manually per the instructions here:
> 
> http://wiki.apache.org/solr/SolrCloud
> 
> Here's the exact URL I'm using:
> 
> http://asu-solr-cloud.elasticbeanstalk.com/admin/collections?action=CREATE&name=directory&numShards=2&replicationFactor=2&maxShardsPerNode=2
> 
> I'm testing out your patch now, and I'll let you know how it goes.
> 
> Thanks for all the help!
> 
> -Al
> 
> On Jun 18, 2013, at 6:47 AM, Erick Erickson wrote:
> 
>> OK, I think I see what's happening. If you do
>> NOT specify an instanceDir on the create
>> (and I'm doing this via the core admin
>> interface, not SolrJ) then the default is
>> used, but not persisted. If you _do_
>> specify the instance dir, it will be persisted.
>> 
>> I've put up another quick patch (tested
>> only in my test case, running full suite
>> now). Can you give it a whirl? You'll have
>> to apply the patch over top of the current
>> 4x, een though the patch is for trunk it
>> applied to 4x cleanly for me and the tests ran.
>> 
>> Thanks,
>> Erick
>> 
>> On Tue, Jun 18, 2013 at 9:02 AM, Erick Erickson <erickerickson@gmail.com> wrote:
>>> OK, I put up a very preliminary patch attached to the bug
>>> if you want to try it out that addresses the extra junk being
>>> put in the <core> tag. Doesn't address the instanceDir issue
>>> since I haven't reproduced it yet.
>>> 
>>> Erick
>>> 
>>> On Tue, Jun 18, 2013 at 8:46 AM, Erick Erickson <erickerickson@gmail.com>
wrote:
>>>> Whoa! What's this junk?
>>>> qt="/admin/cores" wt="javabin" version="2
>>>> 
>>>> That shouldn't be being preserved, and the instancedir should be!
>>>> 
>>>> So I'm guessing you're using SolrJ to create the core, but I just
>>>> reproduced the problem (at least the 'wt="json" ') bit from the
>>>> browser and even from one of my internal tests when I added
>>>> extra parameters.
>>>> 
>>>> That said, instanceDir is being preserved in my test, so I'm not
>>>> seeing everything you're seeing, could you cut/paste your
>>>> create code? I'll see if I can set up a test case for SolrJ to catch
>>>> this too.
>>>> 
>>>> See SOLR-4935
>>>> 
>>>> Thanks for reporting!
>>>> 
>>>> On Mon, Jun 17, 2013 at 5:39 PM, Al Wold <alwold@alwold.com> wrote:
>>>>> Hi Erick,
>>>>> I tried out your changes from the branch_4x branch. It looks good in
terms of preserving the zkHost, but I'm running into an exception because it isn't persisting
the instanceDir attribute on the <core> element.
>>>>> 
>>>>> I've got a few other things I need to take care of, but as soon as I
have time I'll dig in and see if I can figure out what's going on, and see what changed to
make this not work.
>>>>> 
>>>>> Here are details on what the files looked like before/after CREATE call:
>>>>> 
>>>>> original solr.xml:
>>>>> 
>>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>> <solr persistent="true" sharedLib="lib" zkHost="10.116.249.136:2181">
>>>>> <!-- this 8080 might need to change in production -->
>>>>> <cores adminPath="/admin/cores" zkClientTimeout="20000" hostPort="8080"
hostContext="/"/>
>>>>> </solr>
>>>>> 
>>>>> here's what was produced with 4.3 branch + a quick mod to preserve zkHost:
>>>>> 
>>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>> <solr persistent="true" zkHost="10.116.249.136:2181" sharedLib="lib">
>>>>> <cores adminPath="/admin/cores" zkClientTimeout="20000" hostPort="8080"
hostContext="/">
>>>>>   <core loadOnStartup="true" shard="shard1" instanceDir="directory_shard1_replica1/"
transient="false" name="directory_shard1_replica1" collection="directory"/>
>>>>>   <core loadOnStartup="true" shard="shard2" instanceDir="directory_shard2_replica1/"
transient="false" name="directory_shard2_replica1" collection="directory"/>
>>>>> </cores>
>>>>> </solr>
>>>>> 
>>>>> here's what was produced with branch_4x 4.4-SNAPSHOT:
>>>>> 
>>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>> <solr persistent="true" zkHost="10.116.249.136:2181" sharedLib="lib">
>>>>> <cores adminPath="/admin/cores" zkClientTimeout="20000" distribUpdateSoTimeout="0"
distribUpdateConnTimeout="0" hostPort="8080" hostContext="/">
>>>>>   <core shard="shard1" numShards="2" name="directory_shard1_replica2"
collection="directory" qt="/admin/cores" wt="javabin" version="2"/>
>>>>>   <core shard="shard2" numShards="2" name="directory_shard2_replica2"
collection="directory" qt="/admin/cores" wt="javabin" version="2"/>
>>>>> </cores>
>>>>> </solr>
>>>>> 
>>>>> and here's the error from solr.log after restarting after the CREATE:
>>>>> 
>>>>> 2013-06-17 21:37:07,083 1874 [pool-2-thread-1] ERROR org.apache.solr.core.CoreContainer
 - null:java.lang.NullPointerException: Missing required 'instanceDir'
>>>>>       at org.apache.solr.core.CoreDescriptor.doInit(CoreDescriptor.java:133)
>>>>>       at org.apache.solr.core.CoreDescriptor.<init>(CoreDescriptor.java:87)
>>>>>       at org.apache.solr.core.CoreContainer.load(CoreContainer.java:365)
>>>>>       at org.apache.solr.core.CoreContainer.load(CoreContainer.java:221)
>>>>>       at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:190)
>>>>>       at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:124)
>>>>>       at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277)
>>>>>       at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258)
>>>>>       at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382)
>>>>>       at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103)
>>>>>       at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638)
>>>>>       at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294)
>>>>>       at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
>>>>>       at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
>>>>>       at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
>>>>>       at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
>>>>>       at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1099)
>>>>>       at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1621)
>>>>>       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>>>       at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>>>       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>       at java.lang.Thread.run(Thread.java:679)
>>>>> 
>>>>> 
>>>>> On Jun 16, 2013, at 5:38 AM, Erick Erickson wrote:
>>>>> 
>>>>>> Al:
>>>>>> 
>>>>>> As it happens, I hope sometime today to put up a patch for SOLR-4910
>>>>>> that should harden up many things in persisting solr.xml, I'll be
sure
>>>>>> to include this. It's kind of a pain to create an automated test
for
>>>>>> this, so I'll give it a whirl manually.
>>>>>> 
>>>>>> As you say, most of this is going away in 5.0, but it needs to work
for 4.x.
>>>>>> 
>>>>>> And when I get the patch up, if you could give it a "real world"
try
>>>>>> it'd be great!
>>>>>> 
>>>>>> Thanks,
>>>>>> Erick
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 6:15 PM, Al Wold <alwold@alwold.com>
wrote:
>>>>>>> Hi,
>>>>>>> I'm working on setting up a solr cloud test environment, and
the target environment I need to put it in has multiple webapps per tomcat instance. With
that in mind, I wanted/had to avoid putting any configs in system properties. I tried putting
the zkHost in solr.xml, like this:
>>>>>>> 
>>>>>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>>>>> <solr persistent="true" sharedLib="lib" zkHost="10.116.249.136:2181">
>>>>>>>> <!-- this 8080 might need to change in production -->
>>>>>>>> <cores adminPath="/admin/cores" zkClientTimeout="20000"
hostPort="8080" hostContext="/"/>
>>>>>>>> </solr>
>>>>>>> 
>>>>>>> Everything works fine when I first start things up, create collections,
upload docs, search, etc. Creating the collection, however, modifies the solr.xml file, and
doesn't keep the zkHost setting:
>>>>>>> 
>>>>>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>>>>> <solr persistent="true" sharedLib="lib">
>>>>>>>> <cores adminPath="/admin/cores" zkClientTimeout="20000"
hostPort="8080" hostContext="/">
>>>>>>>>  <core loadOnStartup="true" shard="shard2" instanceDir="directory_shard2_replica1/"
transient="false" name="directory_shard2_replica1" collection="directory"/>
>>>>>>>>  <core loadOnStartup="true" shard="shard1" instanceDir="directory_shard1_replica1/"
transient="false" name="directory_shard1_replica1" collection="directory"/>
>>>>>>>> </cores>
>>>>>>>> </solr>
>>>>>>> 
>>>>>>> 
>>>>>>> With that in mind, once I restart tomcat, it no longer knows
it's supposed to be talking to zookeeper, so it looks for local configs and blows up.
>>>>>>> 
>>>>>>> I traced this back to the code in CoreContainer.java, in the
method persistFile(), where it seems to contain no code to write out the zkHost when it updates
solr.xml. I upped the logging on my solr instance to verify this code is executing, so I'm
pretty sure it's the right spot.
>>>>>>> 
>>>>>>> Is anyone else using zkHost in their solr.xml successfully? I
can't see how it would work given this problem.
>>>>>>> 
>>>>>>> Does this seem like a bug? If so, I can probably file a report
and submit a patch. It seems like this problem may become a non-issue in 5.0, based on comments
in the code and some of the discussion in JIRA, but I'm not sure how far off that is.
>>>>>>> 
>>>>>>> Thanks!
>>>>>>> 
>>>>>>> -Al Wold
>>>>>>> 
>>>>> 
>> 
> 


Mime
View raw message