qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
Date Fri, 07 Jul 2017 15:42:00 GMT

    [ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078270#comment-16078270
] 

ASF GitHub Bot commented on DISPATCH-767:
-----------------------------------------

Github user alanconway commented on a diff in the pull request:

    https://github.com/apache/qpid-dispatch/pull/172#discussion_r126161603
  
    --- Diff: src/message.c ---
    @@ -996,21 +1094,47 @@ qd_message_t *qd_message_receive(pn_delivery_t *delivery)
         }
     
         //
    +    // The discard flag indicates if we should continue receiving the message.
    +    // This is pertinent in the case of large messages. When large messages are being
received, we try to send out part of the
    +    // message that has been received so far. If we not able to send it anywhere, there
is no need to keep creating buffers
    +    //
    +    bool discard = qd_message_is_discard((qd_message_t*)msg);
    +
    +    //
         // Get a reference to the tail buffer on the message.  This is the buffer into which
    -    // we will store incoming message data.  If there is no buffer in the message, allocate
    -    // an empty one and add it to the message.
    +    // we will store incoming message data.  If there is no buffer in the message, this
is the
    +    // first time we are here and we need to allocate an empty one and add it to the
message.
         //
    -    buf = DEQ_TAIL(msg->content->buffers);
    -    if (!buf) {
    -        buf = qd_buffer();
    -        DEQ_INSERT_TAIL(msg->content->buffers, buf);
    +    if (!discard) {
    +        buf = DEQ_TAIL(msg->content->buffers);
    +        if (!buf) {
    +            buf = qd_buffer();
    +            DEQ_INSERT_TAIL(msg->content->buffers, buf);
    +        }
         }
     
         while (1) {
    -        //
    -        // Try to receive enough data to fill the remaining space in the tail buffer.
    -        //
    -        rc = pn_link_recv(link, (char*) qd_buffer_cursor(buf), qd_buffer_capacity(buf));
    +        if (discard) {
    +            char dummy[BUFFER_SIZE];
    --- End diff --
    
    Pity that we have to copy out data only to throw it away, but I think you are correct
that we do need to do it. Maybe the AMQP spec has something we can use to abort an incoming
message, but even if it does this code path is probably needed as a fall-back.


> Message Cut-Through/Streaming for efficient handling of large messages
> ----------------------------------------------------------------------
>
>                 Key: DISPATCH-767
>                 URL: https://issues.apache.org/jira/browse/DISPATCH-767
>             Project: Qpid Dispatch
>          Issue Type: Improvement
>          Components: Router Node
>            Reporter: Ted Ross
>            Assignee: Ganesh Murthy
>             Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no need to wait
for the entire message to arrive before starting to send it onward.
> This feature causes the router to route the first frame and allow subsequent frames in
a delivery to be streamed out in pipeline fashion.  Ideally, the memory usage in the router
should only involve pending frames.  This would allow the router to handle arbitrary numbers
of concurrent arbitrarily large messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message