lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: JDK 9 b148 including a refresh of the module system is available on java.net
Date Fri, 09 Dec 2016 17:02:22 GMT
Hi,

 

Sorry my message was incorrect:

There are 2 bugs in Lucene and Java:

-          In Lucene the unmap hack’s exception handling falls over the stupid InaccessibleObjectException
does not extend ReflectiveOperationException issue (I complained about that several times).
I can fix this, but it’s horrible.

-          Java now hides internal of ByteBuffer completely, although the JEP about Unsafe
& Co. said which Java APIs are excluded from the lock down. Next to Unsafe, also ByteBuffer
was on the list of “still allowed APIs where the Java 9 backwards layer helps”. MappedByteBuffer
unmapping is one of the most important parts in Lucene that must be supported, otherwise it
is impossible to use Lucene, Solr, or Elasticsearch with Java 9 in a performant way. We can
also not require from our users to add strange command line options to their JVMs.

 

Uwe

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <http://www.thetaphi.de/> http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: Uwe Schindler [mailto:uwe@thetaphi.de] 
Sent: Friday, December 9, 2016 5:53 PM
To: dev@lucene.apache.org; 'Rory O'Donnell' <rory.odonnell@oracle.com>
Cc: 'Dalibor Topic' <dalibor.topic@oracle.com>; balchandra.vaidya@oracle.com; abdul.kolarkunnu@oracle.com;
'Dawid Weiss' <dawid.weiss@cs.put.poznan.pl>
Subject: RE: JDK 9 b148 including a refresh of the module system is available on java.net

 

Hi,

 

build 148 broke the unmap hack. The problem is an additional bug in Java: The unmap hack should
be more unspecific on the Exception Type (InaccessibleObjectException does not extend ReflectiveOperationException).
I will change this in an issue, but the general problem persists:

 

With the buil 148 version we can no longer unmap byte buffers:

 

java.lang.reflect.InaccessibleObjectException: Unable to make public jdk.internal.ref.Cleaner
java.nio.DirectByteBuffer.cleaner() accessible: module java.base does not "opens java.nio"
to unnamed module @10b73cf

   [junit4]    >       at java.base/jdk.internal.reflect.Reflection.throwInaccessibleObjectException(Reflection.java:427)

   [junit4]    >       at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:201)

   [junit4]    >       at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:192)

   [junit4]    >       at java.base/java.lang.reflect.Method.setAccessible(Method.java:186)

   [junit4]    >       at org.apache.lucene.store.MMapDirectory.unmapHackImpl(MMapDirectory.java:345)

   [junit4]    >       at java.base/java.security.AccessController.doPrivileged(Native
Method)

   [junit4]    >       at org.apache.lucene.store.MMapDirectory.<clinit>(MMapDirectory.java:326)

   [junit4]    >       ... 42 moreThrowable #2: java.lang.NullPointerException

   [junit4]    >       at org.apache.lucene.search.TestDateSort.tearDown(TestDateSort.java:71)

   [junit4]    >       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

   [junit4]    >       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

   [junit4]    >       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

   [junit4]    >       at java.base/java.lang.reflect.Method.invoke(Method.java:538)

   [junit4]    >       at java.base/java.lang.Thread.run(Thread.java:844)

 

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/> 

eMail:  <mailto:uwe@thetaphi.de> uwe@thetaphi.de

 

From: Uwe Schindler [ <mailto:uwe@thetaphi.de> mailto:uwe@thetaphi.de] 
Sent: Friday, December 9, 2016 4:35 PM
To: 'Rory O'Donnell' < <mailto:rory.odonnell@oracle.com> rory.odonnell@oracle.com>
Cc: 'Dalibor Topic' < <mailto:dalibor.topic@oracle.com> dalibor.topic@oracle.com>;
 <mailto:balchandra.vaidya@oracle.com> balchandra.vaidya@oracle.com;  <mailto:abdul.kolarkunnu@oracle.com>
abdul.kolarkunnu@oracle.com; 'Dawid Weiss' < <mailto:dawid.weiss@cs.put.poznan.pl>
dawid.weiss@cs.put.poznan.pl>;  <mailto:dev@lucene.apache.org> dev@lucene.apache.org
Subject: RE: JDK 9 b148 including a refresh of the module system is available on java.net

 

Hi Rory,

 

thanks for the message! I put build 148 in the Jenkins test rotation. Let’s see what happens.

 

Uwe

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/> 

eMail:  <mailto:uwe@thetaphi.de> uwe@thetaphi.de

 

From: Rory O'Donnell [ <mailto:rory.odonnell@oracle.com> mailto:rory.odonnell@oracle.com]

Sent: Friday, December 9, 2016 11:34 AM
To:  <mailto:uwe@thetaphi.de> uwe@thetaphi.de
Cc:  <mailto:rory.odonnell@oracle.com> rory.odonnell@oracle.com; Dalibor Topic <
<mailto:dalibor.topic@oracle.com> dalibor.topic@oracle.com>;  <mailto:balchandra.vaidya@oracle.com>
balchandra.vaidya@oracle.com;  <mailto:abdul.kolarkunnu@oracle.com> abdul.kolarkunnu@oracle.com;
Dawid Weiss < <mailto:dawid.weiss@cs.put.poznan.pl> dawid.weiss@cs.put.poznan.pl>;
 <mailto:dev@lucene.apache.org> dev@lucene.apache.org
Subject: JDK 9 b148 including a refresh of the module system is available on java.net

 

 

Hi Uwe & Dawid, 


JDK 9 build b148 <https://jdk9.java.net/download/>  includes an important Refresh of
the module system [1] , summary of  changes are listed here  <http://download.java.net/java/jdk9/changes/jdk-9+148.html>
. 

This refresh includes a disruptive change that is important to understand. 

For those that have been trying out modules with regular JDK 9 builds then be aware that `requires
public` changes to `requires transitive`. In addition, the binary representation of the module
declaration (module-info.class) has changed so that you need to recompile any modules that
were compiled with previous JDK 9 builds. 

As things stand today in JDK 9 then you use setAccessible to break into non-public elements
of any type in exported packages. However, it cannot be used to break into any type in non-exported
package. The current specified behavior was a compromise for the initial integration of the
module system. It is of course not very satisfactory, hence the #AwkwardStrongEncapsulation
issue [2] on the JSR 376 issues list. With the updated proposal in the JSR, this refresh changes
setAccessible further so that it cannot be used to break into non-public types, or non-public
elements of public types, in exported packages. Code that uses setAccessible to hack into
the private constructor of java.lang.invoke.MethodHandles.Lookup will be disappointed for
example.

This change will expose hacks in many existing libraries and tools. As a workaround then a
new command line option `--add-opens` can be used to open specific packages for "deep reflection".
For example, a really popular build tool fails with this refresh because it uses setAccessible
+ core reflection to hack into a private field of an unmodifiable collection so that it can
mutate it, facepalm! This code will continue to work as before when run with `--add-opens
java.base/java.util=ALL-UNNAMED` to open the package java.util in module java.base to "all
unnamed modules" (think class path).

Any help reporting issues to popular tools and libraries would be appreciated. 

A debugging aid that is useful to identify issues is to run with -Dsun.reflect.debugModuleAccessChecks=true
to get a stack trace when setAccessible fails, this is particularly useful when code swallows
exceptions without any logging.


Rgds,Rory


[1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-November/005276.html <http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html>

[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation

-- 
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland 

Mime
View raw message