lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joel Bernstein (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-8709) Account for out-of-order version numbers in the TopicStream
Date Tue, 05 Apr 2016 16:13:25 GMT

    [ https://issues.apache.org/jira/browse/SOLR-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226550#comment-15226550
] 

Joel Bernstein commented on SOLR-8709:
--------------------------------------

Commit settings were:

1 second hard commits
2 second soft commits

The softCommit is what matters because the TopicStream is working off of index snapshots.

In reviewing the code, it certainly looks like there is room for out-of-order version numbers
to cross commit boundaries, but I would expect to see it happen in a stress test.

I'm thinking about changing this ticket to maintain a checksum of the version numbers for
a time window. Then resending the time window if the checksums don't match. This is much more
manageable then attempting to track all version numbers for a time window. This won't provide
guaranteed *one time* delivery of documents, but it will provide guaranteed delivery of all
documents in a topic, with a reasonable expectation of one time delivery.

> Account for out-of-order version numbers in the TopicStream
> -----------------------------------------------------------
>
>                 Key: SOLR-8709
>                 URL: https://issues.apache.org/jira/browse/SOLR-8709
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Joel Bernstein
>
> Currently the TopicStream can miss documents if version numbers are received out-of-order.
The TopicStream sorts on version number so it will only miss out-of-order versions that span
commit boundaries.
> In order to resolve this issue we can adopt an approach that keeps a set of the last
N version numbers sent for each Topic.  As the documents are scanned we can check for documents
within this time window that do not appear in the sent set. These documents can then be sent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message