qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Wall (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (QPID-8238) [Broker-J] Improve performance of asynchronous publishing of transient messages into topic exchange having queues bound using non-overlapping selectors
Date Tue, 11 Sep 2018 07:53:00 GMT

    [ https://issues.apache.org/jira/browse/QPID-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610224#comment-16610224
] 

Keith Wall edited comment on QPID-8238 at 9/11/18 7:52 AM:
-----------------------------------------------------------

I think it would be better to separate the FieldTable optimisation and any LOGBACK-1027 workaround
performance improvements into separate JIRAs.  This would make it easier for reviewers to
review and users to evaluate the risk of taking a change (or not).   At the moment the FieldTable
changes which affects *all* 0-x users are hidden under a Jira title that talks of topic exchange.
  Conversely, the Jira body talks of a 0-9 performance degradation yet the code change to
the topic exchange impacts a shared 0-9, 0-10, 1.0 path.  I would suggest reverting the 
[c30e8ae|https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=c30e8ae] and  [8c58e38|https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=8c58e38] from
this JIRA.

On the {{FieldTable}} changes, I did like the idea expressed above to make the FieldTables
immutable.  I was imagining turning FieldTable into an interface with two implementations.
 One that accepts the ByteBuffer and one that adapts a Java Map.  The latter would be used
in the BDB upgraders and the conversion layer.   I think this approach would make the implementation
tidier and easier to reason about.  Separately, the idea to make the decoding of the short
strings within the FieldTable on-demand sounds appealing.  This would benefit any application
that was setting many string properties on a message but only using a subset for routing.
 Did you do any testing?   (I did wonder if the approach would also bring benefit to the
strings in BasicContentHeaderProperties - but that would be a separate optimisation).

On the topic exchange changes - could you add some commentary to help explain your thinking
behind this change to allow me to review efficiently.

The turbo filter changes looked reasonable.

 

 


was (Author: k-wall):
I think it would be better to separate the FieldTable optimisation and any LOGBACK-1027 workaround
performance improvements into separate JIRAs.  This would make it easier for reviewers to
review and users to evaluate the risk of taking a change (or not).   At the moment the FieldTable
changes which affects *all* 0-x users are hidden under a Jira title that talks of topic exchange.
  Conversely, the Jira body talks of a 0-9 performance degradation yet the code change to
the topic exchange impacts a shared 0-9, 0-10, 1.0 path.  I would suggest reverting the 
[c30e8ae|https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=c30e8ae] and  [8c58e38|https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=8c58e38] from
this JIRA.

On the {{FieldTable}} changes, I did like the idea expressed above to make the FieldTables
immutable.  I was imagining turning FieldTable into an interface with two implementations.
 One that accepts the ByteBuffer and one that adapts a Java Map.  The latter would be used
in the BDB upgraders and the conversion layer.   I think this approach would make the implementation
tidier and easier to reason about.  Separately, the idea to make the decoding of the short
strings within the FieldTable on-demand sounds appealing.  This would benefit any application
that was setting many string properties on a message but only using a subset for routing.
 Did you do any testing?   (I did wonder if the approach would also bring benefit to the
strings in BasicContentHeaderProperties - but that would be a separate optimisation).

On the topic exchange changes - could you add some commentary to help explain your thinking
behind this change to allow me to review efficiently.

The turbo filter changes looked reasonable.

 

 

 

 

 

> [Broker-J] Improve performance of asynchronous publishing of transient messages into
topic exchange having queues bound using non-overlapping selectors 
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: QPID-8238
>                 URL: https://issues.apache.org/jira/browse/QPID-8238
>             Project: Qpid
>          Issue Type: Improvement
>          Components: Broker-J
>    Affects Versions: qpid-java-broker-7.0.6
>            Reporter: Alex Rudyy
>            Priority: Major
>             Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.7
>
>         Attachments: 0001-QPID-8238-Use-java.lang.String-for-keys-and-values-i.patch
>
>
> The performance of asynchronous publishing of transient messages into topic exchange
which routes messages into queues bound using non-overlapping selectors is 2-3 times slower
than performance of 0.32 broker. The performance degradation is observed with AMQP 0.9, though,
I suspect that the AMQP 0-10 protocol could be affected as well.
> I was running tests with 10 concurrent producers publishing messages  on separate connections
using the same routing key into 10 different queues (subscribers queues) bound to the exchange
using non-overlapping selectors.
> My testing showed that performance of 7.0 broker for this particular use case was 2-3
times worse than performance of 0.32 broker.
> The following factors contributed to degradation of performance:
> •	Copying data from direct memory into heap memory whilst decoding message headers.
Due to this factor, the decoding of message headers is around twice slower. It seems it contributes
around 70% to total performance degradation
> •	The message routing algorithm is slower due to need to support a new feature to route
messages into bound exchanges (in addition to queues) using replacement routing key.
> •	AMQ short strings caching contributes 5-10% to total performance degradation. The
caching was added to manage heap space more efficiently.
> The numbers provided here could be inaccurate due instrumentation overhead whilst profiling
the issue.
> Potentially, caching can be turned off but that will not improve performance much.
> On other hand, adding of additional caching of strings to amqp-short-strings would improve
the performance a bit. Whilst evaluating selectors, the fields used in selector expressions
are represented as java strings but they get converted every time into amqp-short-strings
when looking up for message header values. If 10 queues are bound to the exchange using the
same binding key, the selector expression is evaluated 10 times for the incoming message.
Thus, all selector field names are get converted into amqp-short-strings 10 times as well.
It seems adding caching here can improve the performance. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message