tomee-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leonardo K. Shikida" <shik...@gmail.com>
Subject Re: need advice for single queue configuration
Date Wed, 14 Oct 2015 14:40:23 GMT
Hi

thanks guys for the brainstorm

to make things easier, let me help with the assumptions here

1. it's a single web app
2. this web app uses a single queue
3. the web app itself is the producer (it produces bursts of messages as
result of UI interaction -> someone clicks on a button and hundreds of
messages are sent to the queue)
4. the web app itself is the consumer (target here is to maximize the
number of simultaneous MDBs processing each message, I am considering the
idea of hundreds, if not thousands of MDBs in the pool)
5. it's ok to set the prefetch to 1 per MDB, makes more sense to me, it's
less efficient but it's easier to track
6. I am using producerFlowControl=false
7. trying to use asyncSend=true but it's confusing in activeMQ exactly
WHERE the parameter should go


TIA

Leo

[]

Leo

On Tue, Oct 13, 2015 at 6:23 PM, Andy Gumbrecht <agumbrecht@tomitribe.com>
wrote:

>
> OK, I'm probably jumping in too far with my own assumptions here. So I'll
> try and see the bigger picture. There are TomEE limitations, but we can
> configure them to suit your needs. Bear with me.
>
> If the server is purely dedicated to this purpose then it is fine to load
> it up in this way. Just as long as you know it's going to starve the
> 'thread' pool for all MDBs of all deployed applications. Again, it is fine
> if there is only one app on this server. I'm used to servers running
> multiple MDB operations over multiple applications. If one app consumes the
> entire MDB thread pool then it's game over for the other apps.
>
> That is probably where I am getting crossed wires in my assumptions. The
> difference between the mq resource thread pool, and bean pools. So let me
> explain...
>
> The default MDB adapter is configured with a 'thread' pool of 30 for all
> MDBs:
>
> Default\ JMS\ Resource\ Adapter.ThreadPoolSize=30
>
> So this is going to limit the number of threads that can work for a
> blocking message. Hence, ensuring that the total number of MDBs are limited
> to this figure (minus one just allows a finished bean to dispose and a new
> one to be created, which is not necessary but I just like to have it that
> way).
>
> Now 'bean pools' are the number of actual instances of each MDB type fed
> by the adapter thread pool:
>
> Default\ MDB\ Container.InstanceLimit=10
>
> So if you only have one MDB type on the server then you could set
> InstanceLimit to 29 and be done with it (or increase/decrease those numbers
> to tune for your load)
>
> However, there are more options:
>
> One is as previously described - Annotate your MDB to limit consumption
> using @ActivationConfigProperty. The other is to configure containers for
> each MDB interface type
>
> Configure a container per MDB type in the tomee.xml:
>
> <Container id="MyMdbOne" type="MESSAGE">
>     ....
>     InstanceLimit 10
>     messageListenerInterface an.interface.IMyMdbOne
> </Container>
>
> And annotate your MDB with:
> @MessageDriven(messageListenerInterface = an.interface.IMyMdbOne)
>
> Hope this helps resolve any confusion I may have added.
>
> Andy.
>
> PS. Options can be found here:
> http://tomee.apache.org/jms-resources-and-mdb-container.html
>
>
> On 13/10/2015 15:44, Tim Bain wrote:
>
>> Andy,
>>
>> Can you please describe why using a pool of MDBs that each consume one
>> message at a time till they're done is a problem in TomEE?  It's no
>> problem
>> in JMS/ActiveMQ (see below), but if TomEE has non-JMS reasons why TomEE
>> doesn't handle that scenario well (for example, if TomEE intentionally
>> kills and doesn't restart any MDB that doesn't complete after some
>> timeout), that would be good to know.
>>
>> Starving the pool of MDBs doesn't seem like a bad thing if the goal of the
>> pool is to limit the number of messages being simultaneously consumed.
>> Obviously you have to accept the consequences of that decision and it will
>> cause problems in certain scenarios (such as where you have a mix of fast
>> and slow messages and you want to continue processing fast messages even
>> when there's a backlog of slow messages, and the fact that your KahaDB
>> data
>> files might stay around longer than necessary if you're using KahaDB), but
>> there are other JMS/ActiveMQ constructs available to address those
>> situations (e.g. priority, selectors).  Acking a message on initial
>> receipt
>> and then resending it if processing fails is definitely not the
>> recommended
>> JMS approach, because it modifies the message order on failure (failed
>> messages go to the back of the line on a resend, but the front on a
>> redelivery) and you will lose the message entirely if the client crashes
>> between the ack and the resend.  Instead, use individual ack mode to defer
>> acknowledging the message until you've processed it successfully.  And if
>> you let JMS do the redelivery, it will track the redelivery count
>> automatically.
>>
>> I'm not saying that a single message consumer feeding a work manager is a
>> bad paradigm, just that it's not the only one from a JMS perspective, so
>> unless TomEE has a limitation that makes it not work (and again, if that's
>> the case, please share details), I don't see a reason to tell Leonardo
>> that
>> he's using an invalid approach.
>>
>> Tim
>> Ouch, I'll try and point you in a better direction. A message consumer
>> should never block longer than at maximum a few seconds, else you are
>> going
>> to starve the pool in no time. You should only be checking that the
>> message
>> payload is valid. If you need a long running task then hand the message
>> off
>> to a workmanager. If the task fails, then simply resend the message to the
>> queue and let the next consumer try again. If you do that then be sure to
>> add some message property to keep count of how many times it's failed, so
>> the task doesn't keep sending it.
>>
>> Andy.
>>
>> agumbrecht@tomitribe.com
>> http://www.tomitribe.com
>>
>> Sent from my mobile device.
>>
>> ----- Reply message -----
>> From: "Tim Bain" <tbain@alumni.duke.edu>
>> To: "ActiveMQ Users" <users@activemq.apache.org>
>> Cc: <users@tomee.apache.org>
>> Subject: need advice for single queue configuration
>> Date: Fri, Oct 9, 2015 15:20
>>
>> If you want redelivery to other consumers rather than just the one to
>> which
>> the message was first delivered (which is how I interpreted your paragraph
>> about message-driven beans) and you can live with out-of-order delivery,
>> I'd think you'd want to configure broker redelivery as described at the
>> bottom of
>> http://activemq.apache.org/message-redelivery-and-dlq-handling.html.
>>
>> When TomEE "simply stops consuming the queue", have you taken a thread
>> dump
>> to see what its threads are doing?  Is it possible that every MDB is in
>> the
>> middle of processing a 1-hour message?
>>
>> Have you used a JMX viewer such as JConsole to examine the subscriptions
>> on
>> your destination on the broker when TomEE is having problems, to see if
>> it's still subscribed?
>>
>> Tim
>> On Sep 11, 2015 11:34 AM, "Leonardo K. Shikida" <shikida@gmail.com>
>> wrote:
>>
>> Hi
>>>
>>> I need a simple queue.
>>>
>>> This queue will be persistent and it will use a database in the backend
>>> to
>>> store the messages.
>>>
>>> It will receive produced messages small in size (just one or two string
>>> attributes) in bursts of 10~100.
>>>
>>> It will consume messages by a pool of 100~300 message driven beans
>>>
>> (tomee).
>>
>>> Each message will be consumed and will trigger a long-running job that
>>> may
>>> take from 1 minute to 1 hour to complete. If the job fails, I want the
>>> message to return to the queue so it can be consumed again, no matter how
>>> many retries.
>>>
>>> I don't want to send invalid messages to a "poisoned messages queue".
>>>
>>> Architecture is simple. The queue will run in a single activeMQ instance
>>> and consumers are MDBs from a single TomEE+ web application.
>>>
>>> So, here's what I have so far
>>>
>>> activemq.xml=========================================
>>>
>>>      <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="localhost" dataDirectory="${activemq.data}">
>>>          <destinationPolicy>
>>>              <policyMap>
>>>                  <policyEntries>
>>>                      <policyEntry topic=">" producerFlowControl="true">
>>>                          <pendingMessageLimitStrategy>
>>>                              <constantPendingMessageLimitStrategy
>>> limit="1000" />
>>>                          </pendingMessageLimitStrategy>
>>>                      </policyEntry>
>>>                      <policyEntry queue=">" producerFlowControl="true"
>>> memoryLimit="1mb">
>>>                      </policyEntry>
>>>                  </policyEntries>
>>>              </policyMap>
>>>          </destinationPolicy>
>>>
>>>          <persistenceAdapter>
>>>              <jdbcPersistenceAdapter dataSource="#oracle-ds" />
>>>          </persistenceAdapter>
>>>
>>>          <systemUsage>
>>>              <systemUsage>
>>>                  <memoryUsage>
>>>                      <memoryUsage limit="1 gb" />
>>>                  </memoryUsage>
>>>                  <storeUsage>
>>>                      <storeUsage limit="100 gb" />
>>>                  </storeUsage>
>>>                  <tempUsage>
>>>                      <tempUsage limit="50 gb" />
>>>                  </tempUsage>
>>>              </systemUsage>
>>>          </systemUsage>
>>>
>>>          <transportConnectors>
>>>              <transportConnector name="tcp" uri="tcp://0.0.0.0:61616" />
>>>          </transportConnectors>
>>>      </broker>
>>>
>>>      <bean id="oracle-ds" class="org.apache.commons.dbcp.BasicDataSource"
>>> destroy-method="close">
>>>          <property name="driverClassName"
>>> value="oracle.jdbc.OracleDriver"
>>> />
>>>          <property name="url" value="jdbc:oracle:thin:@localhost
>>> :1521:XE"
>>> />
>>>          <property name="username" value="*****" />
>>>          <property name="password" value="******" />
>>>          <property name="poolPreparedStatements" value="true" />
>>>          <property name="maxActive" value="100" />
>>>      </bean>
>>>
>>> tomee.xml=========================================
>>>
>>>      <Resource id="Default JMS Resource Adapter"
>>> type="ActiveMQResourceAdapter">
>>>          BrokerXmlConfig =  broker:(tcp://localhost:61616)
>>>          ServerUrl = tcp://localhost:61616
>>>          ThreadPoolSize = 200
>>>      </Resource>
>>>
>>>      <Resource id="MyJmsConnectionFactory"
>>> type="javax.jms.ConnectionFactory">
>>>      ResourceAdapter = Default\ JMS\ Resource\ Adapter
>>>      PoolMaxSize = 200
>>>      </Resource>
>>>
>>>      <Resource id="JobQueue" type="javax.jms.Queue" />
>>>
>>> MDB=========================================
>>>
>>> @MessageDriven(activationConfig = {
>>>          @ActivationConfigProperty(propertyName = "destinationType",
>>> propertyValue = "javax.jms.Queue"),
>>>          @ActivationConfigProperty(propertyName = "destination",
>>> propertyValue = "JobQueue"),
>>>          @ActivationConfigProperty(propertyName =
>>> "maxMessagesPerSessions",
>>> propertyValue = "1"),
>>>          @ActivationConfigProperty(propertyName = "maxSessions",
>>> propertyValue = "200"),
>>>          @ActivationConfigProperty(propertyName = "acknowledgeMode",
>>> propertyValue = "Auto-acknowledge") })
>>>
>>> =========================================
>>>
>>> With this configuration, I am still facing some interruptions.
>>>
>>> Sometimes, TomEE simply stops consuming the queue (once or twice per
>>>
>> week),
>>
>>> so I have to restart it. I am still checking the logs for anything
>>> suspicious.
>>>
>>> I am using TomEE+ 1.6.0 stable (quite old, I know) and a standalone AMQ
>>> 5.10 stable in a linux environment and oracle JDK 7.
>>>
>>> But I'd like some advice on how to properly set up activemq / tomee in
>>>
>> this
>>
>>> scenario (that's why I am sending this email to both activemq and tomee
>>> lists)
>>>
>>> Thank you very much, any help is welcome.
>>>
>>> []
>>>
>>> Leo
>>>
>>>
> --
>   Andy Gumbrecht
>   https://twitter.com/AndyGeeDe
>   http://www.tomitribe.com
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message