lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Blum (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-6760) New optimized DistributedQueue implementation for overseer
Date Wed, 19 Aug 2015 16:25:45 GMT

    [ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703277#comment-14703277
] 

Scott Blum edited comment on SOLR-6760 at 8/19/15 4:25 PM:
-----------------------------------------------------------

Fixed.  To recap IRC discussion, the old test code assumed that calling offer() would result
in the offered element immediately being available from poll().  This is contrary to the design
decision in the new DQ.

1) Fixed the test code assumptions and generally cleaned up the test code to be more clear.
2) Documented offer() semantics.

On the decision to decouple offer() and poll():

The rationale for the design decision is that being the offering VM is not "special".  In
the general case different VMs often offer while a single VM is doing the polling, the two
are decoupled in reality.  So there's no real need to guarantee that offer() followed by poll()
in a single VM always returns the element immediately



was (Author: dragonsinth):
Fixed.  To recap IRC discussion, the old test code assumed that calling offer() would result
in the offered element immediately being available from poll().  This is contrary to the design
decision in the new DQ.

1) Fixed the test code assumptions and generally cleaned up the test code to be more clear.
2) Documented offer() semantics.

> New optimized DistributedQueue implementation for overseer
> ----------------------------------------------------------
>
>                 Key: SOLR-6760
>                 URL: https://issues.apache.org/jira/browse/SOLR-6760
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Noble Paul
>            Assignee: Shalin Shekhar Mangar
>         Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, deadlock.patch
>
>
> Currently the DQ works as follows
> * read all items in the directory
> * sort them all 
> * take the head and return it and discard everything else
> * rinse and repeat
> This works well when we have only a handful of items in the Queue. If the items in the
queue is much larger (in tens of thousands) , this is counterproductive
> As the overseer queue is a multiple producers + single consumer queue, We can read them
all in bulk  and before processing each item , just do a zk.exists(itemname) and if all is
well we don't need to do the fetch all + sort thing again



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message