qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kim van der Riet (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (QPID-5924) [linearstore] Qpidd Will Not Start with Large Number of Queues
Date Wed, 06 Aug 2014 19:32:13 GMT

     [ https://issues.apache.org/jira/browse/QPID-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kim van der Riet resolved QPID-5924.
------------------------------------

    Resolution: Fixed

> [linearstore] Qpidd Will Not Start with Large Number of Queues
> --------------------------------------------------------------
>
>                 Key: QPID-5924
>                 URL: https://issues.apache.org/jira/browse/QPID-5924
>             Project: Qpid
>          Issue Type: Bug
>          Components: C++ Broker
>    Affects Versions: 0.22
>         Environment: qpid-cpp-server-0.22-42
> qpid-cpp-server-linearstore-0.22-42
>            Reporter: Brian Bouterse
>            Assignee: Kim van der Riet
>            Priority: Critical
>             Fix For: 0.29
>
>         Attachments: QPID-5924.diff
>
>
> Pulp is an open source project that uses Qpidd. Pulp has need for a large number of queues
10K+, and these queues need to be durable. When creating a large number of queues (thousands),
if you restart qpidd, it won't start. Here is how to reproduce:
> 1. Install qpid-cpp-server and qpid-cpp-server-store
> 2. Start qpidd
> 3. Create a crazy number of unique queues (10K) with durability
> 4. Restart Qpidd
> 5. Observe an error message such as the following
> Starting Qpid AMQP daemon: Daemon startup failed: Queue pulp.agent.5752dc04-7536-4e5c-b406-a0cd5d9c9119:
recoverMessages() failed: jexception 0x0104 RecoveryManager::getFile() threw JERR__FILEIO:
File read or write failure. (/var/lib/qpidd/qls/jrnl/pulp.agent.5752dc04-7536-4e5c-b406-a0cd5d9c9119/818fa4b0-3319-4478-b2b0-d2195f90f695.jrnl)
(/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/linearstore/MessageStoreImpl.cpp:1004)
> Looking at /var/lib/qpidd/qls/jrnl/ directory there is 2676 jrnl files, 2640 of them
start with pulp.agent. In our case the lots of queues are named 'pulp.agent.<UUID>'.
> The expected behavior is that qpidd would start and run awesome with a crazy number of
queues (1 Million +).
> Raising the number of file descriptors is a viable workaround, but eventually those will
run out too. It would be an architectural win if a constant number of file descriptors were
used that are not affected by the number of queues qpidd is managing.
> Perhaps this could be introduced as a new journal type that would run slower but be more
scalable. It could be introduced as qpid-cpp-server-crazy-scalable-but-slower-store.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message