lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
Date Wed, 19 Jan 2011 17:10:45 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983798#action_12983798
] 

Michael McCandless commented on LUCENE-2871:
--------------------------------------------

I attempted to benchmark this patch, testing indexing time for Wikipedia docs, but I hit this
exception:

{noformat}
Exception in thread "Lucene Merge Thread #49" org.apache.lucene.index.MergePolicy$MergeException:
java.nio.BufferOverflowException
	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:507)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:472)
Caused by: java.nio.BufferOverflowException
	at java.nio.Buffer.nextPutIndex(Buffer.java:501)
	at java.nio.DirectByteBuffer.putLong(DirectByteBuffer.java:745)
	at org.apache.lucene.store.FSDirectory$FSChannelIndexOutput.writeLong(FSDirectory.java:706)
	at org.apache.lucene.index.codecs.PrefixCodedTermsWriter.writeTrailer(PrefixCodedTermsWriter.java:146)
	at org.apache.lucene.index.codecs.PrefixCodedTermsWriter.close(PrefixCodedTermsWriter.java:129)
	at org.apache.lucene.index.PerFieldCodecWrapper$FieldsWriter.close(PerFieldCodecWrapper.java:80)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:559)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:120)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3084)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2768)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:379)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:447)
{noformat}

This was after adding ~6 M docs... the 100K doc index built just fine.  Shay any ideas?

> Use FileChannel in FSDirectory
> ------------------------------
>
>                 Key: LUCENE-2871
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2871
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Shay Banon
>         Attachments: LUCENE-2871.patch
>
>
> Explore using FileChannel in FSDirectory to see if it improves write operations performance

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message