lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-6386) TestIndexWriterForceMerge still unreliable in NIGHTLY
Date Thu, 02 Apr 2015 08:09:52 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392344#comment-14392344
] 

Michael McCandless commented on LUCENE-6386:
--------------------------------------------

This is hard to think about :)

In your example I'm assuming "result" is just the same as "level2", but converted into CFS
(ie, s14 really should have been s13.cfs, but I'll just call it s14 here to try not to lose
my sanity).

So, at the exact moment when IW finishes building the final (level3) CFS [s14], you have:

  * original source segments -> 1X

  * level 1 segments -> 1X

  * level 2 segment [s13] non-CFS -> 1X

  * result segment [s14] CFS -> 1X

We do NOT delete level 1 segments after merging to the level 2 non-CFS segment before creating
the result s14 (we used to do this, but it caused complexity/problems because a non-CFS file
can unexpectedly "sneak into" an IW commit or an NRT reader even when you demanded CFS).

So at that moment, the peak temp disk usage is 3X?

> TestIndexWriterForceMerge still unreliable in NIGHTLY
> -----------------------------------------------------
>
>                 Key: LUCENE-6386
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6386
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Robert Muir
>            Assignee: Michael McCandless
>             Fix For: Trunk, 5.1
>
>         Attachments: LUCENE-6386.patch
>
>
> Discovered by ryan beasting (trunk):
> ant test -Dtestcase=TestIndexWriterForceMerge -Dtests.method=testForceMergeTempSpaceUsage
-Dtests.seed=DC9ADB74850A581B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
-Dtests.locale=sr__#Latn -Dtests.timezone=Indian/Chagos -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> {noformat}
>   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterForceMerge
-Dtests.method=testForceMergeTempSpaceUsage -Dtests.seed=DC9ADB74850A581B -Dtests.multiplier=2
-Dtests.nightly=true -Dtests.slow=true -Dtests.locale=sr__#Latn -Dtests.timezone=Indian/Chagos
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] FAILURE 1.20s | TestIndexWriterForceMerge.testForceMergeTempSpaceUsage <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError: forceMerge used too much
temporary space: starting usage was 291570 bytes; final usage was 262469 bytes; max temp usage
was 1079501 but should have been 874710 (= 3X starting usage), BEFORE=
>    [junit4]    > _u.scf              146329
>    [junit4]    > _u.si               635
>    [junit4]    >  |- (inside compound file) _u.fld              2214
>    [junit4]    >  |- (inside compound file) _u.inf              392
>    [junit4]    >  |- (inside compound file) _u.len              2381
>    [junit4]    >  |- (inside compound file) _u.pst              36758
>    [junit4]    >  |- (inside compound file) _u.vec              104144
>    [junit4]    > _s.pst              1338
>    [junit4]    > _s.inf              392
>    [junit4]    > _s.fld              94
>    [junit4]    > _s.len              221
>    [junit4]    > _s.vec              3744
>    [junit4]    > _s.si               624
>    [junit4]    > _t.fld              94
>    [junit4]    > _t.len              221
>    [junit4]    > _t.pst              1338
>    [junit4]    > _t.inf              392
>    [junit4]    > _t.vec              3744
>    [junit4]    > _t.si               624
>    [junit4]    > _v.fld              94
>    [junit4]    > _v.pst              1338
>    [junit4]    > _v.inf              392
>    [junit4]    > _v.vec              3744
>    [junit4]    > _v.si               624
>    [junit4]    > _v.len              221
>    [junit4]    > _w.len              221
>    [junit4]    > _w.pst              1338
>    [junit4]    > _w.inf              392
>    [junit4]    > _w.fld              94
>    [junit4]    > _w.si               624
>    [junit4]    > _w.vec              3744
>    [junit4]    > _x.vec              3744
>    [junit4]    > _x.inf              392
>    [junit4]    > _x.pst              1338
>    [junit4]    > _x.fld              94
>    [junit4]    > _x.si               624
>    [junit4]    > _x.len              221
>    [junit4]    > _y.fld              94
>    [junit4]    > _y.pst              1338
>    [junit4]    > _y.inf              392
>    [junit4]    > _y.si               624
>    [junit4]    > _y.vec              3744
>    [junit4]    > _y.len              221
>    [junit4]    > _z.fld              94
>    [junit4]    > _z.pst              1338
>    [junit4]    > _z.inf              392
>    [junit4]    > _z.len              221
>    [junit4]    > _z.vec              3744
>    [junit4]    > _z.si               624
>    [junit4]    > _10.si              630
>    [junit4]    > _10.fld             94
>    [junit4]    > _10.pst             1338
>    [junit4]    > _10.inf             392
>    [junit4]    > _10.vec             3744
>    [junit4]    > _10.len             221
>    [junit4]    > _11.len             221
>    [junit4]    > _11.si              630
>    [junit4]    > _11.vec             3744
>    [junit4]    > _11.pst             1338
>    [junit4]    > _11.inf             392
>    [junit4]    > _11.fld             94
>    [junit4]    > _12.vec             3744
>    [junit4]    > _12.si              630
>    [junit4]    > _12.len             221
>    [junit4]    > _12.fld             94
>    [junit4]    > _12.pst             1338
>    [junit4]    > _12.inf             392
>    [junit4]    > _13.fld             94
>    [junit4]    > _13.vec             3744
>    [junit4]    > _13.si              630
>    [junit4]    > _13.pst             1338
>    [junit4]    > _13.inf             392
>    [junit4]    > _13.len             221
>    [junit4]    > _14.fld             94
>    [junit4]    > _14.pst             1338
>    [junit4]    > _14.inf             392
>    [junit4]    > _14.si              630
>    [junit4]    > _14.vec             3744
>    [junit4]    > _14.len             221
>    [junit4]    > _15.len             221
>    [junit4]    > _15.vec             3744
>    [junit4]    > _15.si              630
>    [junit4]    > _15.pst             1338
>    [junit4]    > _15.inf             392
>    [junit4]    > _15.fld             94
>    [junit4]    > _16.vec             3744
>    [junit4]    > _16.len             221
>    [junit4]    > _16.fld             94
>    [junit4]    > _16.si              630
>    [junit4]    > _16.pst             1338
>    [junit4]    > _16.inf             392
>    [junit4]    > _17.vec             3744
>    [junit4]    > _17.pst             1338
>    [junit4]    > _17.inf             392
>    [junit4]    > _17.len             221
>    [junit4]    > _17.si              630
>    [junit4]    > _17.fld             94
>    [junit4]    > _18.pst             1338
>    [junit4]    > _18.inf             392
>    [junit4]    > _18.len             221
>    [junit4]    > _18.vec             3744
>    [junit4]    > _18.si              630
>    [junit4]    > _18.fld             94
>    [junit4]    > _19.fld             94
>    [junit4]    > _19.si              630
>    [junit4]    > _19.len             221
>    [junit4]    > _19.vec             3744
>    [junit4]    > _19.pst             1338
>    [junit4]    > _19.inf             392
>    [junit4]    > _1a.fld             94
>    [junit4]    > _1a.pst             1338
>    [junit4]    > _1a.inf             392
>    [junit4]    > _1a.len             221
>    [junit4]    > _1a.vec             3744
>    [junit4]    > _1a.si              630
>    [junit4]    > _1b.fld             94
>    [junit4]    > _1b.si              630
>    [junit4]    > _1b.pst             1338
>    [junit4]    > _1b.inf             392
>    [junit4]    > _1b.vec             3744
>    [junit4]    > _1b.len             221
>    [junit4]    > _1c.vec             3744
>    [junit4]    > _1c.pst             1338
>    [junit4]    > _1c.inf             392
>    [junit4]    > _1c.fld             94
>    [junit4]    > _1c.len             221
>    [junit4]    > _1c.si              630
>    [junit4]    > _1d.pst             1338
>    [junit4]    > _1d.inf             392
>    [junit4]    > _1d.fld             94
>    [junit4]    > _1d.len             221
>    [junit4]    > _1d.vec             3744
>    [junit4]    > _1d.si              630
>    [junit4]    > _1e.fld             94
>    [junit4]    > _1e.pst             1338
>    [junit4]    > _1e.inf             392
>    [junit4]    > _1e.len             221
>    [junit4]    > _1e.vec             3744
>    [junit4]    > _1e.si              630
>    [junit4]    > _1f.fld             40
>    [junit4]    > _1f.pst             195
>    [junit4]    > _1f.inf             392
>    [junit4]    > _1f.len             149
>    [junit4]    > _1f.vec             405
>    [junit4]    > _1f.si              629
>    [junit4]    > AFTER=
>    [junit4]    > _1j.si              635
>    [junit4]    > _1j.scf             261701
>    [junit4]    >  |- (inside compound file) _1j.fld             3982
>    [junit4]    >  |- (inside compound file) _1j.inf             392
>    [junit4]    >  |- (inside compound file) _1j.len             4149
>    [junit4]    >  |- (inside compound file) _1j.pst             66151
>    [junit4]    >  |- (inside compound file) _1j.vec             186577
>    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DC9ADB74850A581B:C6581887EB1ABBDC]:0)
>    [junit4]    > 	at org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage(TestIndexWriterForceMerge.java:199)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=false,coord=no):
{id=IB SPL-D2, content=DFR I(F)B1}, locale=sr__#Latn, timezone=Indian/Chagos
>    [junit4]   2> NOTE: Linux 3.13.0-45-generic amd64/Oracle Corporation 1.8.0_40-ea
(64-bit)/cpus=8,threads=1,free=208092000,total=253231104
>    [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterForceMerge]
>    [junit4] Completed in 1.54s, 1 test, 1 failure <<< FAILURES!
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message