httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: [users@httpd] mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does not understand this error code: proxy: read response failed
Date Sat, 31 Oct 2009 22:46:25 GMT
Damian Traverso wrote:
> Hi,
> we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
> connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
> are experiencing some issues with high CPU load on the Tomcat side,
> and Apache starts logging errors like these
> 
> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
> expired: ajp_ilink_receive() can't receive header
> [Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive failed
> [Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
> this error code: proxy: read response failed from
> 
> On Tomcat side we are using the APR Native Connector, with this settings:
> 
>  <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
> maxThreads="130" backlog="10" connectionTimeout="30000"
> keepAliveTimeout="30000" enableLookups="false" />
> 
> Apache MPM conf:
> 
> ServerLimit 100
> <IfModule mpm_worker_module>
>    StartServers         30
>    MaxClients         2500
>    MinSpareThreads      50
>    MaxSpareThreads     100
>    ThreadsPerChild      25
>    MaxRequestsPerChild   0
> </IfModule>
> 
> mod_proxy_ajp conf:
> 
> <Proxy balancer://jogacraque>
>    BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app01 ttl=30 max=2
>    BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app02 ttl=30 max=2
>    BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app03 ttl=30 max=2
> </Proxy>
> 
> Apache specs:
> 
> Server version: Apache/2.2.13 (Unix)
> Server built:   Sep 29 2009 11:12:35
> Server's Module Magic Number: 20051115:23
> Server loaded:  APR 1.3.8, APR-Util 1.3.9
> Compiled using: APR 1.3.8, APR-Util 1.3.9
> Architecture:   32-bit
> Server MPM:     Worker
>  threaded:     yes (fixed thread count)
>  forked:     yes (variable process count)
> 
> 
> I've beeng testing different settings for mod_proxy_ajp, but I am not
> sure what is the exact relation between the max option for mod_proxy
> and ThreadsPerChild. Should I set max to 1? I understand I would have
> as many connections to each Tomcat box as
> max*(MaxClients/ThreadsPerChild), is this correct?
> 
> BTW, is the total number of connections that mod_proxy_ajp will
> forward to Tomcat related with MaxThreads ? I don't fully understand
> how APR connector works in this case.
> 
Hi.
No great specialist here, but I'll go by the numbers above and give you 
my interpretation.

MaxClients 2500
on your webserver/balancer indicates how many requests you are willing 
to accept and process at the same time, at the Apache level.
ThreadsPerChild 25
means that, if you ever reach these 2500 simultaneous requests, Apache 
will create 100 children processes to handle them, since each child is 
limited to 25 threads, and 2500 / 25 threads -> 100 children.
(Which seems to match the ServerLimit. Good.)

It is so with the mod_jk connector, and probably also with mod_proy_ajp, 
that the connector will automatically calculate how many connections to 
make to Tomcat, based on the number of threads/processes above.

At the Tomcat level, the AJP Connector says
maxThreads="130"
and you have 3 Tomcats, thus for a total maximum number of threads of 
130 X 3 = 390.
One Tomcat thread processes one request at a time.
So the Tomcats, together, can process 390 requests at a time.

There seems to be a mismatch between the number of requests you accept 
in Apache (2500), and the number of threads available to process them in 
Tomcat (390). At the limit, you could have 390 requests being processed 
in Tomcat, and 2110 waiting to get a free Tomcat thread.

That is probably what you are seeing with this log message :
 > [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
 > expired: ajp_ilink_receive() can't receive header

In other words, probably Apache/mod_proy_ajp got tired of waiting.

On the other hand, you indicate that the poor Tomcats are already 
struggling to cope with the load.  Increasing their number of threads is 
thus unlikely to help much.
So it looks like you need more Tomcats.
Or else reduce the MaxClients in Apache.
Or optimise your Tomcat webapps.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message