lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Steffensen <st...@designware.dk>
Subject Re: How to configure Solr to use ZooKeeper ACLs in order to protect it's content
Date Fri, 20 Mar 2015 11:25:55 GMT
Sorry, I did not follow this mailing-list close enough to detect this 
question. But Dmitry mailed to me privately asking for help, so here I am

Initial steps
* mkdir solr-test
* cd solr-test
* Downloaded solr-5.0.0.zip and unzipped into solr-test folder, so that 
I have solr-test/solr-5.0.0 folder
* cd solr-5.0.0
* export SOLR_HOME=$(pwd)
* Started new/empty ZK at localhost:2181 (sure you can do that)

Setting the VM-params
* export 
SOLR_ZK_PROVIDERS="-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider

-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider"
* export SOLR_ZK_CREDS_AND_ACLS="-DzkDigestUsername=admin-user 
-DzkDigestPassword=admin-password 
-DzkDigestReadonlyUsername=readonly-user 
-DzkDigestReadonlyPassword=readonly-password"

Starting solr, just to have the jar extracted into webapp folder, so 
that I can use the classpath you used
* cd $SOLR_HOME/server
* java -jar start.jar
* CTRL-C to stop again

Bootstrapping (essentially creating the /solr root-node in ZK)
* cd $SOLR_HOME/server
* java $SOLR_ZK_PROVIDERS $SOLR_ZK_CREDS_AND_ACLS -classpath 
"$SOLR_HOME/server/solr-webapp/webapp/WEB-INF/lib/*:$SOLR_HOME/server/lib/ext/*" 
org.apache.solr.cloud.ZkCLI -cmd bootstrap -zkhost localhost:2181/solr 
-solrhome $SOLR_HOME/server/solr

Uploading the config
* cd $SOLR_HOME/server
* java $SOLR_ZK_PROVIDERS $SOLR_ZK_CREDS_AND_ACLS -classpath 
"$SOLR_HOME/server/solr-webapp/webapp/WEB-INF/lib/*:$SOLR_HOME/server/lib/ext/*" 
org.apache.solr.cloud.ZkCLI -zkhost localhost:2181/solr -cmd upconfig 
-confdir 
$SOLR_HOME/server/solr/configsets/data_driven_schema_configs/conf 
-confname gettingstarted_shard1_replica1

Starting Solr node
* cd $SOLR_HOME/server
* java $SOLR_ZK_PROVIDERS $SOLR_ZK_CREDS_AND_ACLS 
-Dsolr.solr.home=$SOLR_HOME/server/solr 
-Dsolr.data.dir=$SOLR_HOME/server/solr/gettingstarted_shard1_replica1 
-Dsolr.log=$SOLR_HOME/server/solr/logs -DzkHost=localhost:2181/solr 
-Djetty.port=8983 -jar start.jar

PROBLEM REPRODUCED!!!

Checking out 5.0.0 source-code to see what is wrong. Finding out that 
you need to set the provider-classess in solr.xml - a Solr-node seems 
not to be able to take the provider-classes from VM-params. When I 
handed over the patch for SOLR-4580, VM-parameters was the only way to 
set providers. The other guys added support for setting it in solr.xml, 
which is a good idea. It seems that at the same time VM-params is not 
any longer supported for Solr-nodes. Do not know it that was intentionally?
Anyway. Added the following to <solrcloud>-section in 
$SOLR_HOME/server/solr/solr.xml
     <str 
name="zkCredentialsProvider">org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider</str>
     <str 
name="zkACLProvider">org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider</str>

Trying to start again (without the SOLR_ZK_PROVIDERS VM-params - because 
they are not used anyway)
* java $SOLR_ZK_CREDS_AND_ACLS -Dsolr.solr.home=$SOLR_HOME/server/solr 
-Dsolr.data.dir=$SOLR_HOME/server/solr/gettingstarted_shard1_replica1 
-Dsolr.log=$SOLR_HOME/server/solr/logs -DzkHost=localhost:2181/solr 
-Djetty.port=8983 -jar start.jar

Viola!!!!

Regards, Per Steffensen

On 19/03/15 15:01, Dmitry Karanfilov wrote:
> Looks like it is still broken.
> The fixed name of system property  zkCredentialsProvider and zkACLProvider
> are only impacted on the zkcli.sh script (org.apache.solr.cloud.ZkCLI).
> So using command bellow, I'm able to *bootstrap *and *upconfig *to the
> Zookeeper with appropriate credentials and ACLs:
>
> export
> SOLR_ZK_PROVIDERS="-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
> -DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider"
> export SOLR_ZK_CREDS_AND_ACLS="-DzkDigestUsername=admin-user
> -DzkDigestPassword=admin-password -DzkDigestReadonlyUsername=readonly-user
> -DzkDigestReadonlyPassword=readonly-password"
>
> java $SOLR_ZK_PROVIDERS $SOLR_ZK_CREDS_AND_ACLS -classpath
> "server/solr-webapp/webapp/WEB-INF/lib/*:server/lib/ext/*"
> org.apache.solr.cloud.ZkCLI -cmd bootstrap -zkhost 10.0.1.112:2181/solr
> -solrhome /opt/solr/example/cloud/node1/solr/
> java $SOLR_ZK_PROVIDERS $SOLR_ZK_CREDS_AND_ACLS -classpath
> "server/solr-webapp/webapp/WEB-INF/lib/*:server/lib/ext/*"
> org.apache.solr.cloud.ZkCLI -zkhost 10.0.1.112:2181/solr -cmd upconfig
> -confdir /opt/solr/server/solr/configsets/data_driven_schema_configs/conf
> -confname gettingstarted_shard1_replica1
>
>
> But when I start a Solr it is not able to connect to the Zookeeper:
>
> java $SOLR_ZK_PROVIDERS $SOLR_ZK_CREDS_AND_ACLS
> -Dsolr.solr.home=/opt/solr/example/cloud/node1/solr
> -Dsolr.data.dir=/opt/solr/example/cloud/node1/solr/gettingstarted_shard1_replica1
> -Dsolr.log=/opt/solr/example/cloud/node1/logs -DzkHost=10.0.1.112:2181/solr
> -Djetty.port=8983 -jar start.jar
>
> Here is logs:
> 0    [main] INFO  org.eclipse.jetty.server.Server  ? jetty-8.1.10.v20130312
> 156  [main] INFO  org.eclipse.jetty.deploy.providers.ScanningAppProvider  ?
> Deployment monitor /opt/solr-5.0.0/server/contexts at interval 0
> 205  [main] INFO  org.eclipse.jetty.deploy.DeploymentManager  ? Deployable
> added: /opt/solr-5.0.0/server/contexts/solr-jetty-context.xml
> 4253 [main] INFO  org.eclipse.jetty.webapp.StandardDescriptorProcessor  ?
> NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
> 4600 [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  ?
> SolrDispatchFilter.init()WebAppClassLoader=2048834776@7a1ebcd8
> 4650 [main] INFO  org.apache.solr.core.SolrResourceLoader  ? JNDI not
> configured for solr (NoInitialContextEx)
> 4651 [main] INFO  org.apache.solr.core.SolrResourceLoader  ? using system
> property solr.solr.home: /opt/solr/example/cloud/node1/solr
> 4657 [main] INFO  org.apache.solr.core.SolrResourceLoader  ? new
> SolrResourceLoader for directory: '/opt/solr/example/cloud/node1/solr/'
> 5305 [main] INFO  org.apache.solr.core.ConfigSolr  ? Loading container
> configuration from /opt/solr/example/cloud/node1/solr/solr.xml
> 5646 [main] INFO  org.apache.solr.core.CoresLocator  ? Config-defined core
> root directory: /opt/solr/example/cloud/node1/solr
> 5677 [main] INFO  org.apache.solr.core.CoreContainer  ? New CoreContainer
> 510147134
> 5682 [main] INFO  org.apache.solr.core.CoreContainer  ? Loading cores into
> CoreContainer [instanceDir=/opt/solr/example/cloud/node1/solr/]
> 5749 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting socketTimeout to: 600000
> 5750 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting urlScheme to: null
> 5760 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting connTimeout to: 60000
> 5761 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting maxConnectionsPerHost to: 20
> 5771 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting maxConnections to: 10000
> 5771 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting corePoolSize to: 0
> 5772 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting maximumPoolSize to: 2147483647
> 5772 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting maxThreadIdleTime to: 5
> 5778 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting sizeOfQueue to: -1
> 5779 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting fairnessPolicy to: false
> 5779 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory
>   ? Setting useRetries to: false
> 6336 [main] INFO  org.apache.solr.update.UpdateShardHandler  ? Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=600000&connTimeout=60000&retry=true
> 6339 [main] INFO  org.apache.solr.logging.LogWatcher  ? SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> 6340 [main] INFO  org.apache.solr.logging.LogWatcher  ? Registering Log
> Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> 6346 [main] INFO  org.apache.solr.core.CoreContainer  ? Host Name:
> 6347 [main] INFO  org.apache.solr.core.ZkContainer  ? Zookeeper client=
> 10.0.1.112:2181/solr7
> 6428 [main] INFO  org.apache.solr.cloud.ZkController  ? zkHost includes
> chroot
> *6430 [main] INFO  org.apache.solr.common.cloud.SolrZkClient  ? Using
> ZkCredentialsProvider:
> org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider*
> 6595 [main] INFO  org.apache.solr.common.cloud.ConnectionManager  ? Waiting
> for client to connect to ZooKeeper
> 6860 [zkCallback-2-thread-1] INFO
>   org.apache.solr.common.cloud.ConnectionManager  ? Watcher
> org.apache.solr.common.cloud.ConnectionManager@33881d3
> name:ZooKeeperConnection Watcher:10.0.1.112:2181 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> 6862 [main] INFO  org.apache.solr.common.cloud.ConnectionManager  ? Client
> is connected to ZooKeeper
> *6863 [main] INFO  org.apache.solr.common.cloud.SolrZkClient  ? Using
> ZkACLProvider:
> org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider*
> 6936 [main] INFO  org.apache.solr.common.cloud.ConnectionManager  ? Waiting
> for client to connect to ZooKeeper
> 6988 [zkCallback-3-thread-1] INFO
>   org.apache.solr.common.cloud.ConnectionManager  ? Watcher
> org.apache.solr.common.cloud.ConnectionManager@570cafbf
> name:ZooKeeperConnection Watcher:10.0.1.112:2181/solr7 got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> 6989 [main] INFO  org.apache.solr.common.cloud.ConnectionManager  ? Client
> is connected to ZooKeeper
> 7079 [main] ERROR org.apache.solr.cloud.Overseer  ? Could not create
> Overseer node
> *org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode =
> NoAuth for /overseer*
>          at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
>          at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>
>     ................................
>
>
>
> Is there someone who have it working?
>
> Thanks,
> Dmitry
>
>
> On 17 March 2015 at 19:12, Dmitry Karanfilov <kdsnice@gmail.com> wrote:
>
>> Sorry, this is CORRECT:
>> SOLR_ZK_PROVIDERS="-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
>> \
>>
>> -DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider"
>>
>> SOLR_ZK_CREDS_AND_ACLS="-DzkDigestUsername=admin-user
>> -DzkDigestPassword=admin-password \
>> -DzkDigestReadonlyUsername=readonly-user
>> -DzkDigestReadonlyPassword=readonly-password"
>>
>> On 17 March 2015 at 18:32, Dmitry Karanfilov <kdsnice@gmail.com> wrote:
>>
>>> I found the issue - it is in documentation:
>>>
>>> WRONG:
>>> SOLR_ZK_PROVIDERS="-DdefaultZkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
>>> \
>>>
>>> -DdefaultZkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider"
>>>
>>> SOLR_ZK_CREDS_AND_ACLS="-DzkDigestUsername=admin-user
>>> -DzkDigestPassword=admin-password \
>>> -DzkDigestReadonlyUsername=readonly-user
>>> -DzkDigestReadonlyPassword=readonly-password"
>>>
>>> CORRECT:
>>> SOLR_ZK_PROVIDERS="-DZkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
>>> \
>>>
>>> -DZkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider"
>>>
>>> SOLR_ZK_CREDS_AND_ACLS="-DzkDigestUsername=admin-user
>>> -DzkDigestPassword=admin-password \
>>> -DzkDigestReadonlyUsername=readonly-user
>>> -DzkDigestReadonlyPassword=readonly-password"
>>>
>>>
>>> On 17 March 2015 at 15:50, Dmitry Karanfilov <kdsnice@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> I'm trying to configure Solr to use  ZooKeeper ACLs as  it is described
>>>> on this wiki page
>>>> https://cwiki.apache.org/confluence/display/solr/ZooKeeper+Access+Control,
>>>> but I have no luck.
>>>> Do I understand correctly that in order to tell Solr about the
>>>> credentials required to access the content in ZooKeeper, I need to compile
>>>> (from source) the
>>>> VMParamsSingleSetCredentialsDigestZkCredentialsProvider.class and
>>>> VMParamsAllAndReadonlyDigestZkACLProvider.class and put them to classpath
>>>> when starting Solr?
>>>> Can someone please provide a step-by-step guide how to do this. I
>>>> googled a lot - there is nothing about it in Internet, the only information
>>>> is wiki page above, but it is unclear.
>>>> I'm fighting with Solr and ZooKeeper ACLs third day.
>>>> Please help!
>>>>
>>>> Thank you,
>>>> Dmitry
>>>>
>>>


Mime
View raw message