qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Bouterse (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (QPID-5637) qpid.messaging Issues With Forking
Date Fri, 25 Apr 2014 18:03:14 GMT

     [ https://issues.apache.org/jira/browse/QPID-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Brian Bouterse updated QPID-5637:
---------------------------------

    Description: 
qpid.messaging has an issue with forking in the following situation.

1.  A parent Python process imports and uses qpid.messaging to connect to a a Qpid broker
2.  The parent process forks a child process
3.  The child process imports qpid.messaging and tries to connect to a Qpid broker.

I expected to see the child process use qpid.messaging normally as it would if it weren't
forked in the way described above.  Instead, the server receives the opening of a TCP socket,
but the client never sends the AMQP protocol announcement.

[Forking bring child descriptors with it|http://man7.org/linux/man-pages/man2/fork.2.html]
 Because file descriptors on the parent and the child are the same, I expect qpid.messaging
to have the parent and child process share the file descriptors and socket and work without
any modification.

There is at least one place where I do understand how this can be avoided.  One of the issues
is that the file descriptors registered by the Selector object inside of qpid.messaging are
stale after the fork.  The Selector object uses a singleton pattern to provide a reference
to the same Selector object no matter how many times you call it.  This selector object already
has registered file descriptors with the filesystem, which allow the selector to read/write
data in an I/O efficient manner.  See the attached patch for an example of this.

This is not the only issue though, even with this patch in place, communication flows, but
queue creation, and deletion seem to fail.  Here is the overall way to reproduce this issue:

1.  clone our fork of kombu:       `git clone git@github.com:pulp/kombu.git`
2.  Change into the kombu folder     `cd kombu`
3.  Switch to the branch containing the qpid code:  `git checkout pulp-dep-3.0.15-with-qpid`
4.  Install kombu onto your system or virtualenv (I do it systemwide using sudo):   `sudo
python setup.py develop`
5.  pip install celery version 3.1.11

  was:
qpid.messaging has an issue with forking in the following situation.

1.  A parent Python process imports and uses qpid.messaging to connect to a a Qpid broker
2.  The parent process forks a child process
3.  The child process imports qpid.messaging and tries to connect to a Qpid broker.

I expected to see the child process use qpid.messaging normally as it would if it weren't
forked in the way described above.  Instead, the server receives the opening of a TCP socket,
but the client never sends the AMQP protocol announcement.

[Forking bring child descriptors with it|http://man7.org/linux/man-pages/man2/fork.2.html]
 Because file descriptors on the parent and the child are the same, I expect qpid.messaging
to have the parent and child process share the file descriptors and socket and work without
any modification.

There is at least one place where I do understand how this can be avoided.  One of the issues
is that the file descriptors registered by the Selector object inside of qpid.messaging are
stale after the fork.  The Selector object uses a singleton pattern to provide a reference
to the same Selector object no matter how many times you call it.  This selector object already
has registered file descriptors with the filesystem, which allow the selector to read/write
data in an I/O efficient manner.  See the attached patch for an example of this.

This is not the only issue though, even with this patch in place, communication flows, but
queue creation, and deletion seem to fail.  Here is the overall way to reproduce this issue:

1.  clone our fork of kombu:       `git clone git@github.com:pulp/kombu.git`
2.  Switch to the branch containing the qpid code:  ``


> qpid.messaging Issues With Forking
> ----------------------------------
>
>                 Key: QPID-5637
>                 URL: https://issues.apache.org/jira/browse/QPID-5637
>             Project: Qpid
>          Issue Type: Bug
>          Components: Python Client
>    Affects Versions: 0.24
>            Reporter: Brian Bouterse
>             Fix For: 0.18, 0.22, 0.24, 0.26, 0.27
>
>         Attachments: 0001-Fix-for-QPID-5637.patch
>
>
> qpid.messaging has an issue with forking in the following situation.
> 1.  A parent Python process imports and uses qpid.messaging to connect to a a Qpid broker
> 2.  The parent process forks a child process
> 3.  The child process imports qpid.messaging and tries to connect to a Qpid broker.
> I expected to see the child process use qpid.messaging normally as it would if it weren't
forked in the way described above.  Instead, the server receives the opening of a TCP socket,
but the client never sends the AMQP protocol announcement.
> [Forking bring child descriptors with it|http://man7.org/linux/man-pages/man2/fork.2.html]
 Because file descriptors on the parent and the child are the same, I expect qpid.messaging
to have the parent and child process share the file descriptors and socket and work without
any modification.
> There is at least one place where I do understand how this can be avoided.  One of the
issues is that the file descriptors registered by the Selector object inside of qpid.messaging
are stale after the fork.  The Selector object uses a singleton pattern to provide a reference
to the same Selector object no matter how many times you call it.  This selector object already
has registered file descriptors with the filesystem, which allow the selector to read/write
data in an I/O efficient manner.  See the attached patch for an example of this.
> This is not the only issue though, even with this patch in place, communication flows,
but queue creation, and deletion seem to fail.  Here is the overall way to reproduce this
issue:
> 1.  clone our fork of kombu:       `git clone git@github.com:pulp/kombu.git`
> 2.  Change into the kombu folder     `cd kombu`
> 3.  Switch to the branch containing the qpid code:  `git checkout pulp-dep-3.0.15-with-qpid`
> 4.  Install kombu onto your system or virtualenv (I do it systemwide using sudo):   `sudo
python setup.py develop`
> 5.  pip install celery version 3.1.11



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message