httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Don Poitras <Don.Poit...@sas.com>
Subject [users@httpd] Reverse proxy server - SIGSEGV
Date Tue, 11 Jun 2019 17:34:49 GMT
Hello. Recently, we ran a stress test (using Loadrunner) against our backend server using Apache
as a reverse-proxy server. We've been running this way for some years. It's not clear when
the last time (or if ever) this stress test was run, but we noticed that we were getting SegVs
in various locations (mainly in mod_proxy). We're running version 2.4.39 of httpd and version
1.7.0 of APR. We're also using worker mpm and if we switch to the event or prefork mpms, we
don't get any errors. We were able to reproduce the problem at 2.4.38 and 2.4.34 and didn't
try any other versions.

Looking at the core files, we could see corrupted memory chains and sometimes we see a pointer
to thread_mutex_cleanup(). Since this pointer is only referenced in the code as an argument
to apr_pool_cleanup_register(), this led us to believe that we were seeing a case where two
threads were accessing the same pool concurrently. We rebuilt httpd using --enable-pool-concurrency-check
and the SegVs turned into aborts emitted by the pool concurrency check.

Looking at these new core files, it could be seen that two threads were concurrently accessing
the same worker proxy connection pool. While proxy_util.c has one mutex call around the call
to apr_sockaddr_info_get() (where the proxy connection pool is used), there were other calls
in this module that needed the same protection.

I see a couple of outstanding bugs listed where this might be the fix:
https://bz.apache.org/bugzilla/show_bug.cgi?id=58491
https://bz.apache.org/bugzilla/show_bug.cgi?id=58529
https://bz.apache.org/bugzilla/show_bug.cgi?id=62357

We have a fix that works and would like to submit this to the bug database. Let me know if
that seems reasonable, or if there are any questions that we can answer here before taking
that step. Thanks.

-- 
Don Poitras - Host R&D    SAS Institute Inc.            SAS Campus Drive 
mailto:sasdtp@sas.com     (919)531-5637  Fax:677-4444   Cary, NC 27513


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message