httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mick Sheppard" <Mick.Shepp...@cpp.co.uk>
Subject [users@httpd] Large DBM ssl_scache causing performance issues, possible DOS
Date Tue, 05 May 2009 11:12:39 GMT
Hi,

 

I've been unable to find any information on the web concerning an issue
we are currently having and want to check that we haven't overlooked any
possible options.

 

We run an instance of Apache 2.2 on Solaris 10 running as both a forward
and reverse proxy for discrete SSL web services transactions. Our
external clients authenticate themselves using SSL client certificates.
We authenticate ourselves to external clients also using SSL client
certificates. Our SSL session cache settings are as follows:

 

SSLSessionCache         dbm:/usr/local/apache2/logs/ssl_scache

SSLSessionCacheTimeout  300

 

We have seen increasing traffic over the past few weeks and observed a
periodic slowdown in response time from the Apache server. A restart of
the server has cleared the problem but it always came back after time.
Using dtrace we were able to observe that a single httpd process was
clocking up tens of thousands of reads in a short period of time, these
were associated with the file /usr/local/apache2/logs/ssl_scache.pag.
This file was quite large, over 1 GB, and the number of reads
approximated to the file size/1024.

 

Looking at the debug output of Apache we could see that it was checking
whether to expire entries from this file. Looking at the code each httpd
process does at least one sequential read through the file every
SSLSessionCacheTimeout. Whilst performing this read the SSL mutex is
locked. Under load these scans appear to coincide for each child process
with the result that the last client to gain the mutex will have been
blocked for (time-to-scan-cache * nclients) seconds. This severely
hampers the ability of our server to service requests.

 

As our web services traffic is in the form of discrete operations we
don't see that we have a need for the SSL session cache at all. Each
request will be services by the initial client process so the other
client processes have no need of the information. Consequently we are
looking to change the SSLSessionCache setting to none to disable it. 

 

We suspect that we are seeing this problem because of the use of client
certificates and the fact that they seem to be stored in the cache.
Based on our observations it would appear that if someone has a valid
client cert then, with the ssl_scache configured in this manner, they
can effectively perform a denial of service on the apache web server.

 

Our questions are whether anyone has come across this before and how
they solved it, and also if there is will be any unforeseen result of us
turning off the session cache?

 

 

Mick Sheppard

Enterprise Systems Support Analyst

CPP Group PLC

Holgate Park

York

YO26 4GA

Tel: 01904 544562

Fax:  01904 544683

Web: www.cpp.co.uk <http://www.cpp.co.uk/> 

Corporate: www.cppgroupplc.com <blocked::http://www.cppgroupplc.com/> 

 


This is an email from the CPP Group Plc, Holgate Park, York, YO26 4GA; telephone +44 (0)1904
544500.
This message may contain information that is confidential. If you are not the intended recipient,
you may not peruse, use, disseminate, distribute or copy this message. If you have received
this
message in error, please notify the sender immediately by email, facsimile or telephone and
either
return or destroy the original message.

The CPP Group Plc accepts no responsibility for any changes made to this message after it
has been
sent by the original author.  This email has been scanned for all viruses by the MessageLabs

Email Security System.

Mime
View raw message