lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4482) Likely Zing JVM bug causes failures in TestPayloadNearQuery
Date Sat, 13 Oct 2012 17:42:02 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475671#comment-13475671
] 

Uwe Schindler commented on LUCENE-4482:
---------------------------------------

bq. We don't test on VirtualBOX, so I don't know for sure. In general, Zing works fine when
run on top of hypervisors that fully support things like 2MB page mappings (the same sort
of support needed for hugetlb feature to work). Unfortunately, there are some hypervisors
out there (e.g. some versions of Xen for paravirt guests) that don't support that, and will
crash a vanilla linux kernel trying to use hugetlb. Zing won't work in such cases either,
and for the same reasons...

In that case, the VM is not running inside a guest OS, but in parallel to a hypervisor using
the same linux kernel. The question was if the 2 modules may conflict to each other. But I
could also imagine to use Zing inside a virtual machine on one of my servers using Lucene
(once the bugs are fixed).

bq. A 4 to 6 order of magnitude difference in pause time and in sustainable allocation rate
is so big that a C4 that uses the vanilla memory management api would be unusable at this
point. Think of the difference between a 20usec phase shift and a 20 second pause...

Have you thought about making this kernel module available to the kernel developers for other
potential use cases (like virtual machines) also needing to re-allocate lots of RAM and influence
paging/unmapping/mapping?

I did not find the GPL source code of your kernel module only the binary downloads. Where
can I get it. It's easy to hook it's build into DKMS (if its a standard Makefile, see https://help.ubuntu.com/community/DKMS)
without any custom debian package.
                
> Likely Zing JVM bug causes failures in TestPayloadNearQuery
> -----------------------------------------------------------
>
>                 Key: LUCENE-4482
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4482
>             Project: Lucene - Core
>          Issue Type: Bug
>         Environment: Lucene trunk, rev 1397735
> Zing:
> {noformat}
>   java version "1.6.0_31"
>   Java(TM) SE Runtime Environment (build 1.6.0_31-6)
>   Java HotSpot(TM) 64-Bit Tiered VM (build 1.6.0_31-ZVM_5.2.3.0-b6-product-azlinuxM-X86_64,
mixed mode)
> {noformat}
> Ubuntu 12.04 LTS 3.2.0-23-generic kernel
>            Reporter: Michael McCandless
>         Attachments: LUCENE-4482.patch
>
>
> I dug into one of the Lucene test failures when running with Zing JVM
> (available free for open source devs...).  At least one other test
> sometimes fails but I haven't dug into that yet.
> I managed to get the failure easily reproduced: with the attached
> patch, on rev 1397735 checkout, if you cd to lucene/core and run:
> {noformat}
>   ant test -Dtests.jvms=1 -Dtests.seed=C3802435F5FB39D0 -Dtests.showSuccess=true
> {noformat}
> Then you'll hit several failures in TestPayloadNearQuery, eg:
> {noformat}
> Suite: org.apache.lucene.search.payloads.TestPayloadNearQuery
>   1> FAILED
>   2> NOTE: reproduce with: ant test  -Dtestcase=TestPayloadNearQuery -Dtests.method=test
-Dtests.seed=C3802435F5FB39D0 -Dtests.slow=true -Dtests.locale=ga -Dtests.timezone=America/Adak
-Dtests.file.encoding=US-ASCII
> ERROR   0.01s | TestPayloadNearQuery.test <<<
>    > Throwable #1: java.lang.RuntimeException: overridden idfExplain method in TestPayloadNearQuery.BoostingSimilarity
was not called
>    > 	at __randomizedtesting.SeedInfo.seed([C3802435F5FB39D0:4BD41BEF5B075428]:0)
>    > 	at org.apache.lucene.search.similarities.TFIDFSimilarity.computeWeight(TFIDFSimilarity.java:740)
>    > 	at org.apache.lucene.search.spans.SpanWeight.<init>(SpanWeight.java:62)
>    > 	at org.apache.lucene.search.payloads.PayloadNearQuery$PayloadNearSpanWeight.<init>(PayloadNearQuery.java:147)
>    > 	at org.apache.lucene.search.payloads.PayloadNearQuery.createWeight(PayloadNearQuery.java:75)
>    > 	at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:648)
>    > 	at org.apache.lucene.search.AssertingIndexSearcher.createNormalizedWeight(AssertingIndexSearcher.java:60)
>    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:265)
>    > 	at org.apache.lucene.search.payloads.TestPayloadNearQuery.test(TestPayloadNearQuery.java:146)
>    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>    > 	at java.lang.reflect.Method.invoke(Method.java:597)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>    > 	at java.lang.Thread.run(Thread.java:661)
> {noformat}
> The patch at least isolates the JVM bug even if it's not exactly a
> minimal test :)  Somehow the idfExplain method, which
> is overridden in this test's BoostingSimilarity, fails to be called
> (the super.idfExplain is called instead), which leads to the test
> failures.
> The failure does not happen if you run this test in isolation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message