lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yuanyun.cn (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-6776) Data lost when use SoftCommit and TLog
Date Mon, 01 Dec 2014 17:21:13 GMT

    [ https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230053#comment-14230053
] 

yuanyun.cn edited comment on SOLR-6776 at 12/1/14 5:20 PM:
-----------------------------------------------------------

The finish of UpdateProcessoris is always called in org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
SolrQueryResponse).

When we add a doc without softcommit or commit,  the org.apache.solr.update.processor.RunUpdateProcessor.finish()
will call getUpdateLog().finish() to fsync the tlog.

But if add a doc with softcommit=true, RunUpdateProcessor.finish() will not call getUpdateLog().finish(),
and will not fsync the tlog.

This is kind of not right.

User enables transaction log for data durability, to make sure there is no data lost.
So I think it should always fsync the tlog after add this doc to solr and before the hard
commit.



was (Author: yuanyun.cn):
The finish of UpdateProcessoris always in org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
SolrQueryResponse).






> Data lost when use SoftCommit and TLog
> --------------------------------------
>
>                 Key: SOLR-6776
>                 URL: https://issues.apache.org/jira/browse/SOLR-6776
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 4.10
>            Reporter: yuanyun.cn
>              Labels: softCommit, updateLog
>             Fix For: 4.10.3
>
>
> We enabled update log and change autoCommit to some bigger value 10 mins.
> After restart, we push one doc with softCommit=true
> http://localhost:8983/solr/update?stream.body=<add><doc><field name="id">id1</field></doc></add>&softCommit=true
> Then we kill the java process after a min. 
> After restart, Tlog failed to replay with following exception, and there is no data in
solr.
> 6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog  รป Failure
to open existing log file (non fatal) E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
> log\tlog.0000000000000000000:org.apache.solr.common.SolrException: java.io.EOFException
>         at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:181)
>         at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
>         at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:134)
>         at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)
>         at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:100)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
>         at java.lang.reflect.Constructor.newInstance(Unknown Source)
>         at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
>         at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:835)
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
>         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
>         at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
>         at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
>         at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
>         at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
>         at org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
>         at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:159)
>         ... 19 more
> Check the code: seems this is related with: org.apache.solr.update.processor.RunUpdateProcessor,
in processCommit, it sets changesSinceCommit=false(even we are using softCommit)
> So in finish, updateLog.finish will not be called.
>   public void finish() throws IOException {
>     if (changesSinceCommit && updateHandler.getUpdateLog() != null) {
>       updateHandler.getUpdateLog().finish(null);
>     }
>     super.finish();
>   }
> To fix this issue: I have to change RunUpdateProcessor.processCommit like below:
>     if (!cmd.softCommit) {
>       changesSinceCommit = false;
>     }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message