tomee-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Gumbrecht <agumbre...@tomitribe.com>
Subject Re: need advice for single queue configuration
Date Tue, 13 Oct 2015 21:23:03 GMT

OK, I'm probably jumping in too far with my own assumptions here. So 
I'll try and see the bigger picture. There are TomEE limitations, but we 
can configure them to suit your needs. Bear with me.

If the server is purely dedicated to this purpose then it is fine to 
load it up in this way. Just as long as you know it's going to starve 
the 'thread' pool for all MDBs of all deployed applications. Again, it 
is fine if there is only one app on this server. I'm used to servers 
running multiple MDB operations over multiple applications. If one app 
consumes the entire MDB thread pool then it's game over for the other apps.

That is probably where I am getting crossed wires in my assumptions. The 
difference between the mq resource thread pool, and bean pools. So let 
me explain...

The default MDB adapter is configured with a 'thread' pool of 30 for all 
MDBs:

Default\ JMS\ Resource\ Adapter.ThreadPoolSize=30

So this is going to limit the number of threads that can work for a 
blocking message. Hence, ensuring that the total number of MDBs are 
limited to this figure (minus one just allows a finished bean to dispose 
and a new one to be created, which is not necessary but I just like to 
have it that way).

Now 'bean pools' are the number of actual instances of each MDB type fed 
by the adapter thread pool:

Default\ MDB\ Container.InstanceLimit=10

So if you only have one MDB type on the server then you could set 
InstanceLimit to 29 and be done with it (or increase/decrease those 
numbers to tune for your load)

However, there are more options:

One is as previously described - Annotate your MDB to limit consumption 
using @ActivationConfigProperty. The other is to configure containers 
for each MDB interface type

Configure a container per MDB type in the tomee.xml:

<Container id="MyMdbOne" type="MESSAGE">
     ....
     InstanceLimit 10
     messageListenerInterface an.interface.IMyMdbOne
</Container>

And annotate your MDB with:
@MessageDriven(messageListenerInterface = an.interface.IMyMdbOne)

Hope this helps resolve any confusion I may have added.

Andy.

PS. Options can be found here: 
http://tomee.apache.org/jms-resources-and-mdb-container.html

On 13/10/2015 15:44, Tim Bain wrote:
> Andy,
>
> Can you please describe why using a pool of MDBs that each consume one
> message at a time till they're done is a problem in TomEE?  It's no problem
> in JMS/ActiveMQ (see below), but if TomEE has non-JMS reasons why TomEE
> doesn't handle that scenario well (for example, if TomEE intentionally
> kills and doesn't restart any MDB that doesn't complete after some
> timeout), that would be good to know.
>
> Starving the pool of MDBs doesn't seem like a bad thing if the goal of the
> pool is to limit the number of messages being simultaneously consumed.
> Obviously you have to accept the consequences of that decision and it will
> cause problems in certain scenarios (such as where you have a mix of fast
> and slow messages and you want to continue processing fast messages even
> when there's a backlog of slow messages, and the fact that your KahaDB data
> files might stay around longer than necessary if you're using KahaDB), but
> there are other JMS/ActiveMQ constructs available to address those
> situations (e.g. priority, selectors).  Acking a message on initial receipt
> and then resending it if processing fails is definitely not the recommended
> JMS approach, because it modifies the message order on failure (failed
> messages go to the back of the line on a resend, but the front on a
> redelivery) and you will lose the message entirely if the client crashes
> between the ack and the resend.  Instead, use individual ack mode to defer
> acknowledging the message until you've processed it successfully.  And if
> you let JMS do the redelivery, it will track the redelivery count
> automatically.
>
> I'm not saying that a single message consumer feeding a work manager is a
> bad paradigm, just that it's not the only one from a JMS perspective, so
> unless TomEE has a limitation that makes it not work (and again, if that's
> the case, please share details), I don't see a reason to tell Leonardo that
> he's using an invalid approach.
>
> Tim
> Ouch, I'll try and point you in a better direction. A message consumer
> should never block longer than at maximum a few seconds, else you are going
> to starve the pool in no time. You should only be checking that the message
> payload is valid. If you need a long running task then hand the message off
> to a workmanager. If the task fails, then simply resend the message to the
> queue and let the next consumer try again. If you do that then be sure to
> add some message property to keep count of how many times it's failed, so
> the task doesn't keep sending it.
>
> Andy.
>
> agumbrecht@tomitribe.com
> http://www.tomitribe.com
>
> Sent from my mobile device.
>
> ----- Reply message -----
> From: "Tim Bain" <tbain@alumni.duke.edu>
> To: "ActiveMQ Users" <users@activemq.apache.org>
> Cc: <users@tomee.apache.org>
> Subject: need advice for single queue configuration
> Date: Fri, Oct 9, 2015 15:20
>
> If you want redelivery to other consumers rather than just the one to which
> the message was first delivered (which is how I interpreted your paragraph
> about message-driven beans) and you can live with out-of-order delivery,
> I'd think you'd want to configure broker redelivery as described at the
> bottom of
> http://activemq.apache.org/message-redelivery-and-dlq-handling.html.
>
> When TomEE "simply stops consuming the queue", have you taken a thread dump
> to see what its threads are doing?  Is it possible that every MDB is in the
> middle of processing a 1-hour message?
>
> Have you used a JMX viewer such as JConsole to examine the subscriptions on
> your destination on the broker when TomEE is having problems, to see if
> it's still subscribed?
>
> Tim
> On Sep 11, 2015 11:34 AM, "Leonardo K. Shikida" <shikida@gmail.com> wrote:
>
>> Hi
>>
>> I need a simple queue.
>>
>> This queue will be persistent and it will use a database in the backend to
>> store the messages.
>>
>> It will receive produced messages small in size (just one or two string
>> attributes) in bursts of 10~100.
>>
>> It will consume messages by a pool of 100~300 message driven beans
> (tomee).
>> Each message will be consumed and will trigger a long-running job that may
>> take from 1 minute to 1 hour to complete. If the job fails, I want the
>> message to return to the queue so it can be consumed again, no matter how
>> many retries.
>>
>> I don't want to send invalid messages to a "poisoned messages queue".
>>
>> Architecture is simple. The queue will run in a single activeMQ instance
>> and consumers are MDBs from a single TomEE+ web application.
>>
>> So, here's what I have so far
>>
>> activemq.xml=========================================
>>
>>      <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="localhost" dataDirectory="${activemq.data}">
>>          <destinationPolicy>
>>              <policyMap>
>>                  <policyEntries>
>>                      <policyEntry topic=">" producerFlowControl="true">
>>                          <pendingMessageLimitStrategy>
>>                              <constantPendingMessageLimitStrategy
>> limit="1000" />
>>                          </pendingMessageLimitStrategy>
>>                      </policyEntry>
>>                      <policyEntry queue=">" producerFlowControl="true"
>> memoryLimit="1mb">
>>                      </policyEntry>
>>                  </policyEntries>
>>              </policyMap>
>>          </destinationPolicy>
>>
>>          <persistenceAdapter>
>>              <jdbcPersistenceAdapter dataSource="#oracle-ds" />
>>          </persistenceAdapter>
>>
>>          <systemUsage>
>>              <systemUsage>
>>                  <memoryUsage>
>>                      <memoryUsage limit="1 gb" />
>>                  </memoryUsage>
>>                  <storeUsage>
>>                      <storeUsage limit="100 gb" />
>>                  </storeUsage>
>>                  <tempUsage>
>>                      <tempUsage limit="50 gb" />
>>                  </tempUsage>
>>              </systemUsage>
>>          </systemUsage>
>>
>>          <transportConnectors>
>>              <transportConnector name="tcp" uri="tcp://0.0.0.0:61616" />
>>          </transportConnectors>
>>      </broker>
>>
>>      <bean id="oracle-ds" class="org.apache.commons.dbcp.BasicDataSource"
>> destroy-method="close">
>>          <property name="driverClassName" value="oracle.jdbc.OracleDriver"
>> />
>>          <property name="url" value="jdbc:oracle:thin:@localhost:1521:XE"
>> />
>>          <property name="username" value="*****" />
>>          <property name="password" value="******" />
>>          <property name="poolPreparedStatements" value="true" />
>>          <property name="maxActive" value="100" />
>>      </bean>
>>
>> tomee.xml=========================================
>>
>>      <Resource id="Default JMS Resource Adapter"
>> type="ActiveMQResourceAdapter">
>>          BrokerXmlConfig =  broker:(tcp://localhost:61616)
>>          ServerUrl = tcp://localhost:61616
>>          ThreadPoolSize = 200
>>      </Resource>
>>
>>      <Resource id="MyJmsConnectionFactory"
>> type="javax.jms.ConnectionFactory">
>>      ResourceAdapter = Default\ JMS\ Resource\ Adapter
>>      PoolMaxSize = 200
>>      </Resource>
>>
>>      <Resource id="JobQueue" type="javax.jms.Queue" />
>>
>> MDB=========================================
>>
>> @MessageDriven(activationConfig = {
>>          @ActivationConfigProperty(propertyName = "destinationType",
>> propertyValue = "javax.jms.Queue"),
>>          @ActivationConfigProperty(propertyName = "destination",
>> propertyValue = "JobQueue"),
>>          @ActivationConfigProperty(propertyName = "maxMessagesPerSessions",
>> propertyValue = "1"),
>>          @ActivationConfigProperty(propertyName = "maxSessions",
>> propertyValue = "200"),
>>          @ActivationConfigProperty(propertyName = "acknowledgeMode",
>> propertyValue = "Auto-acknowledge") })
>>
>> =========================================
>>
>> With this configuration, I am still facing some interruptions.
>>
>> Sometimes, TomEE simply stops consuming the queue (once or twice per
> week),
>> so I have to restart it. I am still checking the logs for anything
>> suspicious.
>>
>> I am using TomEE+ 1.6.0 stable (quite old, I know) and a standalone AMQ
>> 5.10 stable in a linux environment and oracle JDK 7.
>>
>> But I'd like some advice on how to properly set up activemq / tomee in
> this
>> scenario (that's why I am sending this email to both activemq and tomee
>> lists)
>>
>> Thank you very much, any help is welcome.
>>
>> []
>>
>> Leo
>>

-- 
   Andy Gumbrecht
   https://twitter.com/AndyGeeDe
   http://www.tomitribe.com


Mime
View raw message