lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Blum (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-6760) New optimized DistributedQueue implementation for overseer
Date Wed, 19 Aug 2015 16:24:46 GMT

    [ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703277#comment-14703277
] 

Scott Blum commented on SOLR-6760:
----------------------------------

Fixed.  To recap IRC discussion, the old test code assumed that calling offer() would result
in the offered element immediately being available from poll().  This is contrary to the design
decision in the new DQ.

1) Fixed the test code assumptions and generally cleaned up the test code to be more clear.
2) Documented offer() semantics.

> New optimized DistributedQueue implementation for overseer
> ----------------------------------------------------------
>
>                 Key: SOLR-6760
>                 URL: https://issues.apache.org/jira/browse/SOLR-6760
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Noble Paul
>            Assignee: Shalin Shekhar Mangar
>         Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, deadlock.patch
>
>
> Currently the DQ works as follows
> * read all items in the directory
> * sort them all 
> * take the head and return it and discard everything else
> * rinse and repeat
> This works well when we have only a handful of items in the Queue. If the items in the
queue is much larger (in tens of thousands) , this is counterproductive
> As the overseer queue is a multiple producers + single consumer queue, We can read them
all in bulk  and before processing each item , just do a zk.exists(itemname) and if all is
well we don't need to do the fetch all + sort thing again



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message