lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Risden (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-9515) Update to Hadoop 3
Date Thu, 31 Jan 2019 22:09:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757746#comment-16757746
] 

Kevin Risden edited comment on SOLR-9515 at 1/31/19 10:08 PM:
--------------------------------------------------------------

So this caused failures on JDK9+.... not sure how the below is possible currently since solr-tests.policy
([https://github.com/apache/lucene-solr/blob/master/lucene/tools/junit4/solr-tests.policy#L27)] allows
read access to that path.

[https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23605]
{code:java}
[junit4]   2> java.io.IOException: Failed to start sub tasks to add replica in replica
map :java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_B3EBC148FC827CD8-001/tempDir-001/hdfsBaseDir/data/data3/current/BP-669531916-88.99.242.108-1548970895105/current/finalized"
"read")
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.getVolumeMap(BlockPoolSlice.java:439)
~[hadoop-hdfs-3.2.0.jar:?]
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:1003)
~[hadoop-hdfs-3.2.0.jar:?]
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:201)
~[hadoop-hdfs-3.2.0.jar:?]
{code}
Snippet from solr-tests.policy:
{code:java}
permission java.io.FilePermission "<<ALL FILES>>", "read,execute";
permission java.io.FilePermission "${junit4.childvm.cwd}", "read,execute";
permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp", "read,execute,write,delete";
permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp${/}-", "read,execute,write,delete";
permission java.io.FilePermission "${junit4.childvm.cwd}${/}jacoco.db", "write";
permission java.io.FilePermission "${junit4.tempDir}${/}*", "read,execute,write,delete";
permission java.io.FilePermission "${clover.db.dir}${/}-", "read,execute,write,delete";
permission java.io.FilePermission "${tests.linedocsfile}", "read";
permission java.nio.file.LinkPermission "hard";
{code}
Variables from run:
{code:java}
junit4.childvm.cwd=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0
junit.tempDir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/temp
java.security.manager=org.apache.lucene.util.TestSecurityManager
java.security.policy/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/junit4/solr-tests.policy{code}
So we should have read on the paths that HDFS is trying to use?


was (Author: risdenk):
So this caused failures on JDK9+.... not sure how the below is possible currently since solr-tests.policy
([https://github.com/apache/lucene-solr/blob/master/lucene/tools/junit4/solr-tests.policy#L27)] allows
read access to that path.

[https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23605]
{code:java}
[junit4]   2> java.io.IOException: Failed to start sub tasks to add replica in replica
map :java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_B3EBC148FC827CD8-001/tempDir-001/hdfsBaseDir/data/data3/current/BP-669531916-88.99.242.108-1548970895105/current/finalized"
"read")
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.getVolumeMap(BlockPoolSlice.java:439)
~[hadoop-hdfs-3.2.0.jar:?]
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:1003)
~[hadoop-hdfs-3.2.0.jar:?]
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:201)
~[hadoop-hdfs-3.2.0.jar:?]
{code}
Snippet from solr-tests.policy:
{code:java}
permission java.io.FilePermission "<<ALL FILES>>", "read,execute";
permission java.io.FilePermission "${junit4.childvm.cwd}", "read,execute";
permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp", "read,execute,write,delete";
permission java.io.FilePermission "${junit4.childvm.cwd}${/}temp${/}-", "read,execute,write,delete";
permission java.io.FilePermission "${junit4.childvm.cwd}${/}jacoco.db", "write";
permission java.io.FilePermission "${junit4.tempDir}${/}*", "read,execute,write,delete";
permission java.io.FilePermission "${clover.db.dir}${/}-", "read,execute,write,delete";
permission java.io.FilePermission "${tests.linedocsfile}", "read";
permission java.nio.file.LinkPermission "hard";
{code}
Variables from run:
{code:java}
junit4.childvm.cwd=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0
junit.tempDir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/temp{code}
So we should have read on the paths that HDFS is trying to use?

> Update to Hadoop 3
> ------------------
>
>                 Key: SOLR-9515
>                 URL: https://issues.apache.org/jira/browse/SOLR-9515
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Mark Miller
>            Assignee: Kevin Risden
>            Priority: Blocker
>             Fix For: 8.0, master (9.0)
>
>         Attachments: SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch, SOLR-9515.patch,
SOLR-9515.patch, SOLR-9515.patch
>
>          Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Hadoop 3 is not out yet, but I'd like to iron out the upgrade to be prepared. I'll start
up a dev branch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message