lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anshum Gupta <ans...@anshumgupta.net>
Subject Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Date Fri, 20 Nov 2015 23:18:21 GMT
This seems unrelated and more like a user error somewhere. Can you just
follow the steps, without any security settings i.e. not even uploading
security.json and see if you still see this? Sorry, but I don't have access
to the code right now, I try and look at this later tonight.

On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:

> Thank you for opening SOLR-8326
>
> As a side note, in the procedure you listed, even before adding the
> collection-admin-edit authorization, I'm already hitting trouble: stopping
> and restarting a node results in the following
>
> INFO  - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard2_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
> INFO  - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO  - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard1_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
> INFO  - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO  - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO  - 2015-11-20 22:48:41.289; [   ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> INFO  - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO  - 2015-11-20 22:48:41.291; [   ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 25
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
>         at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>         at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>         at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>         at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
>         at org.apache.http.entity.ContentType.parse(ContentType.java:273)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
> INFO  - 2015-11-20 22:48:41.298; [   ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO  - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
> ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
>         at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>         at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>         at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>         at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
>         at org.apache.http.entity.ContentType.parse(ContentType.java:273)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
> ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO  - 2015-11-20 22:48:41.318; [   ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 26
> INFO  - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
>
>
> I would not be surprised if this were to be some unrelated issue (the
> symptoms are quite different)
>
>
>
> Thanks again
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Friday, November 20, 2015 1:31 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Collections API were available before November of 2014, if that is when you
> took the class. However, it was only with Solr 5.0 (released in Feb 2015)
> that the only supported mechanism to create a collection was restricted to
> Collections API.
>
> Here are the list of steps that you'd need to run to see that things are
> fine for you without the read permission:
> * Untar and setup Solr, don't start it yet
> * Start clean zookeeper
> * Put the security.json in zk, without anything other than a security-edit
> permission. Find the content of the file below. Upload it using your own zk
> client or through the solr script:
> > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd putfile /security.json ~/security.json
>
> security.json:
>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
>
> * Start solr:
> > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>
> You would need to key in a few things e.g. #nodes and ports, leave them at
> the default values of 2 nodes and 8983/7574, unless you want to run Solr on
> a different port. Then let it create a default collection to just make sure
> that everything works fine.
>
> * Add the collection-admin-edit command:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" :
> {"name":"collection-admin-edit", "role":"admin"}}'
>
> At this point, everything should be working fine. Restarting the nodes
>  should also work fine. You can try 2 things at this point:
> 1. Create a new collection with 1 shard and 1 replica and then try adding a
> replica, here's how:
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
>
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
>
> This should work fine.
>
> 2. After this, try restarting the solr cluster. Here's how you can do so,
> assuming you didn't change any of the defaults and you are running zk on
> localhost:2181. If not, just change those values below:
> > bin/solr stop -all
>
> After this, check that Solr was actually stopped. I'd also suggest you tail
> the logs on both nodes when they are coming up to see any errors, if any.
> The logs would be here: example/cloud/node1/logs/solr.log
> and example/cloud/node2/logs/solr.log
>
> > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>
> If you get to this checkpoint fine, try adding a read permission.
> Add a permission:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" : {"name":"read",
> "role":"read"}}'
>
> Add a user:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authentication
> -H 'Content-type:application/json' -d '{"set-user" :
> {"solrread":"solrRocks"}}'
>
> Assign a role to the user:
> >curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-user-role" :
> {"solrread":["read"]}}'
>
> After this, you should start having issues with ADDREPLICA.
> Also, as you would at this point have a collection with a shard that has a
> replication factor > 1 (remember the ADDREPLICA we did earlier), you would
> have issues when you restart the cluster again using the steps I mentioned
> above.
>
>
> Can you confirm this? I guess I'll just use this text to create a new JIRA
> now.
>
>
> On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
> > Thank you again for the reply.
> >
> > Below is the Email I was about to send prior to your reply a moment ago:
> > shall I try again without "read" in the security.json?
> >
> >
> >
> > The Collections API method was not discussed in the "Unleashed" class at
> > the conference in DC in 2014 (probably because it was not yet available),
> > so I was using the method I knew.
> >
> > I have now tried again using admin/collections?action=CREATE (using
> > different port numbers to avoid confusion from the failed previous
> > attempts: the previously created nodes had been shutdown and their
> > core.properties files renamed so as not to be discovered), but the
> results
> > are the same:
> > INFO  - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting
> > Replication Recovery.
> > INFO  - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
> > buffering updates.
> > INFO  - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
> > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> > INFO  - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Attempting
> > to replicate from http://
> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
> while
> > trying to
> >
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at http://
> {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> > Expected mime type application/octet-stream but got text/html. <html>
> > <head>
> > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> > <title>Error 401 Unauthorized request, Response code: 401</title>
> > </head>
> > <body><h2>HTTP ERROR 401</h2>
> > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> > <pre>    Unauthorized request, Response code:
> > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
> >
> > </body>
> > </html>
> >
> >         at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> >         at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> >         at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> >         at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> >         at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> >         at
> >
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> >         at
> >
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> >         at
> >
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> >         at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >
> > INFO  - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery
> > failed - trying again... (2)
> > INFO  - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
> > seconds before trying to recover again (3)
> >
> >
> > Below is a list of the steps I took.
> >
> > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json
> > ~/solr/security151119a.json
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> > ../../solr/configsets/basic_configs/conf -confname xmpl3
> > cd ../../../bin/
> > ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> > ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> > '
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> > '
> >
> >
> >
> >
> > Can you provide a list of steps to take in an out-of-the-box directory
> > tree whereby ADDREPLICA _will_ work with security.json already in place?
> >
> >
> >
> >
> > -----Original Message-----
> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> > Sent: Thursday, November 19, 2015 3:44 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >
> > I'll try out what you did later in the day, as soon as I get time but why
> > exactly are you creating cores manually? Seems like you manually create a
> > core and the try to add a replica. Can you try using the Collections API
> to
> > create a collection?
> >
> > Starting Solr 5.0, the only supported way to create a new collection is
> via
> > the Collections API. Creating a core would lead to a collection creation
> > but that's not really supported. It was just something that was done when
> > there were no Collections API.
> >
> >
> > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > craig.oakley@nih.gov> wrote:
> >
> > > I tried again with the following security.json, but the results were
> the
> > > same:
> > >
> > > {
> > >   "authentication":{
> > >     "class":"solr.BasicAuthPlugin",
> > >     "credentials":{
> > >       "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> > >       "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> > >     "":{"v":9}},
> > >   "authorization":{
> > >     "class":"solr.RuleBasedAuthorizationPlugin",
> > >     "user-role":{
> > >       "solr":[
> > >         "admin",
> > >         "read",
> > >         "xmpladmin",
> > >         "xmplgen",
> > >         "xmplsel"],
> > >       "solruser":[
> > >         "read",
> > >         "xmplgen",
> > >         "xmplsel"]},
> > >     "permissions":[
> > >       {
> > >         "name":"security-edit",
> > >         "role":"admin"},
> > >       {
> > >         "name":"xmpl_admin",
> > >         "collection":"xmpl",
> > >         "path":"/admin/*",
> > >         "role":"xmpladmin"},
> > >       {
> > >         "name":"xmpl_sel",
> > >         "collection":"xmpl",
> > >         "path":"/select/*",
> > >         "role":null},
> > >       {
> > >          "name":"all-admin",
> > >          "collection":null,
> > >          "path":"/*",
> > >          "role":"xmplgen"},
> > >       {
> > >          "name":"all-core-handlers",
> > >          "path":"/*",
> > >          "role":"xmplgen"}],
> > >     "":{"v":42}}}
> > >
> > > -----Original Message-----
> > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> > > Sent: Thursday, November 19, 2015 1:46 PM
> > > To: 'solr-user@lucene.apache.org' <solr-user@lucene.apache.org>
> > > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> > >
> > > I note that the thread called "Security Problems" (most recent post by
> > > Nobel Paul) seems like it may help with much of what I'm trying to do.
> I
> > > will see to what extent that may help.
> > >
> >
> >
> >
> > --
> > Anshum Gupta
> >
>
>
>
> --
> Anshum Gupta
>



-- 
Anshum Gupta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message