Looking at our production environment, I see that we use an absolute path in proxy.config.ssl.CA.cert.filename.  Could you try setting that to /ssl/ca.pem?  Perhaps a bug came in with the path manipulation.

On Wed, Jan 6, 2021 at 3:01 PM Zack Bartel <zbartel@proofpoint.com> wrote:
Thanks for the help!
Yes, the client certificate is directly signed by the certificate in ca.pem. There is no intermediary. I did not know about the "ssl" debug flag but this is very helpful. It seems your theory about failing the client cert verification was correct.

[Jan  6 08:21:22.635] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213 (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_WANT_READ (2), errno=0
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1139 (sslServerHandShakeEvent)> (ssl) Go on with the handshake state=5
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510 (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 8193 ret: 1 State: SSLv3/TLS write server done
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:401 (ssl_verify_client_callback)> (ssl) Callback: verify client cert
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1603 (callHooks)> (ssl) callHooks sslHandshakeHookState=5
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1675 (callHooks)> (ssl) callHooks iterated to curHook=(nil)
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510 (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 16392 ret: 592 State: error
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510 (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 8194 ret: -1 State: error
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:2344 (SSLAccept)> (ssl.error.accept) SSL accept returned -1, ssl_error=1, ERR_get_error=337100934 (error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed)
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213 (sslServerHandShakeEvent)> (ssl-diag) SSL::140678460933888:error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed:../ssl/statem/statem_srvr.c:3701: peer address is 192.168.1.4
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213 (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_SSL (1), errno=0
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1339 (sslServerHandShakeEvent)> (ssl-diag) SSLNetVConnection::sslServerHandShakeEvent, SSL_ERROR_SSL errno=0


The certificate is quite old but valid.
        Validity
            Not Before: Aug 18 14:29:55 2017 GMT
            Not After : Aug 16 14:29:55 2027 GMT

There is nothing noteworthy in diags.log nor any startup errors or warnings. 

Thanks again,
Zack



On Jan 6, 2021, at 7:38 AM, Susan Hinrichs <shinrich@verizonmedia.com> wrote:

The config looks good to me.   There are autests that exercise ATS requiring certs from the client with a variety of CA verification configurations.  I haven't worked with the 8.x branch, but those tests are run against that branch I believe.  

Is your client certificate directly signed by the certificate in ca.pem?  Is there an intermediate cert floating around someplace?  Assuming the certs are exchanged as in the working case, it seems likely that the failure is occurring in the client certificate verification.  I assume that ATS is sending the SSL alert.

Any interesting start up failure messages in diags.log?

The next thing to check is turning on debugging and setting the ssl debug tag. 

On Tue, Jan 5, 2021 at 4:49 PM Zack Bartel <zbartel@proofpoint.com> wrote:
Hello, 
We are using client -> TS SSL termination and using client certificates and a CA. While upgrading to 8.x this has stopped working and I'm having trouble figuring out why. I was wondering if there may have been some change in 8.x or error with my config  which would be obvious to someone. 

The error is "SSL alert number 80" internal error.

SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write certificate verify
SSL_connect:SSLv3/TLS write change cipher spec


SSL_connect:SSLv3/TLS write finished
read from 0x564393d04170 [0x564393d0e6a3] (5 bytes => 5 (0x5))
0000 - 15 03 03 00 02                                    .....
read from 0x564393d04170 [0x564393d0e6a8] (2 bytes => 2 (0x2))
0000 - 02 50                                             .P

SSL3 alert read:fatal:internal error
SSL_connect:error in error
139713745085760:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80



I've inspected the traffic with Wireshark to confirm the exchange is identical between the versions up until the "Internal Error" response. Tried up to 7.1.12-rc0 which works and 8.0.0 -> 8.1.x which does not. I've also tried on Ubuntu and Centos8 with the same results.

Relevant config:


records.config:
CONFIG proxy.config.ssl.CA.cert.path STRING /ssl/
CONFIG proxy.config.ssl.CA.cert.filename STRING ca.pem

CONFIG proxy.config.ssl.client.verify.server INT 1
CONFIG proxy.config.ssl.client.certification_level INT 2
CONFIG proxy.config.ssl.client.CA.cert.path STRING /etc/pki/tls/certs/
CONFIG proxy.config.ssl.client.CA.cert.filename STRING ca-bundle.crt

CONFIG proxy.config.ssl.client.cipher_suite STRING TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:DES-CBC3-SHA

CONFIG proxy.config.ssl.server.private_key.path STRING /ssl/
CONFIG proxy.config.ssl.server.cert.path STRING /ssl/

CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

ssl_multicert.config:
ssl_cert_name=server.pem


Any help on this greatly appreciated, thank you,

Zack