qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carsten Lohmann (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DISPATCH-1086) Dispatch Router sporadically goes into a state where TLS connections to the auth service fail
Date Wed, 19 Sep 2018 14:07:00 GMT

    [ https://issues.apache.org/jira/browse/DISPATCH-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620303#comment-16620303
] 

Carsten Lohmann edited comment on DISPATCH-1086 at 9/19/18 2:06 PM:
--------------------------------------------------------------------

Here are backtraces from separate core dumps:

1:
{noformat}
#0  0x00007f4a0578f4fd in pn_list_get (list=0x1424900, index=24140)
    at /build/qpid-proton-src/c/src/core/object/list.c:42
#1  0x00007f4a0579866f in pni_session_bound (ssn=<optimized out>)
    at /build/qpid-proton-src/c/src/core/engine.c:1021
#2  0x00007f4a0579892b in pn_connection_bound (
    connection=connection@entry=0x14244a0)
    at /build/qpid-proton-src/c/src/core/engine.c:157
#3  0x00007f4a0579e268 in pn_transport_bind (transport=0xf1f760,
    connection=0x14244a0) at /build/qpid-proton-src/c/src/core/transport.c:706
#4  0x00007f4a05797d35 in batch_next (batch=0x1461330)
    at /build/qpid-proton-src/c/src/core/connection_driver.c:41
#5  0x00007f4a0557a6f1 in pconnection_batch_next ()
    at /build/qpid-proton-src/c/src/proactor/epoll.c:948
#6  0x00007f4a05a0c2fb in thread_run (arg=arg@entry=0xf0bf60)
    at /build/qpid-dispatch-src/src/server.c:976
#7  0x00007f4a05a0c590 in qd_server_run (qd=<optimized out>)
    at /build/qpid-dispatch-src/src/server.c:1247
#8  0x000000000040182c in main_process (
    config_path=0x7fff809c4965 "/tmp/qdrouterd.conf",
    python_pkgdir=<optimized out>, fd=2)
    at /build/qpid-dispatch-src/router/src/main.c:112
#9  0x0000000000401589 in main (argc=3, argv=0x7fff809c3ba8)
    at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
2:
{noformat}
#0  ssl_cert_dup (cert=0x0) at ssl/ssl_cert.c:89
#1  0x00007fda9b487b98 in SSL_new (ctx=0x1fb8a40) at ssl/ssl_lib.c:716
#2  0x00007fda9c95b0c8 in init_ssl_socket (transport=0x7fda8014ec60,
    ssl=ssl@entry=0x7fda8004dc70)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#3  0x00007fda9c95bc18 in init_ssl_socket (ssl=0x7fda8004dc70,
    transport=0x7fda8014ec60)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#4  process_input_ssl (transport=0x7fda8014ec60, layer=0,
    input_data=0x7fda80163f90 "\220\a", available=0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:963
#5  0x00007fda9c95317a in transport_consume (
    transport=transport@entry=0x7fda8014ec60)
    at /build/qpid-proton-src/c/src/core/transport.c:1821
#6  0x00007fda9c954eda in pn_transport_close_tail (transport=0x7fda8014ec60)
    at /build/qpid-proton-src/c/src/core/transport.c:2972
#7  0x00007fda9c94cdea in pn_connection_driver_read_close (
    d=d@entry=0x7fda800e8968)
    at /build/qpid-proton-src/c/src/core/connection_driver.c:114
#8  0x00007fda9c72f5f8 in pconnection_process (pc=pc@entry=0x7fda800e83c0,
    events=events@entry=0, timeout=timeout@entry=false,
    topup=topup@entry=true, is_io_2=is_io_2@entry=false)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:1230
#9  0x00007fda9c72f754 in pconnection_batch_next ()
    at /build/qpid-proton-src/c/src/proactor/epoll.c:953
#10 0x00007fda9cbc12fb in thread_run (arg=0x1bd0f60)
    at /build/qpid-dispatch-src/src/server.c:976
#11 0x00007fda9c512594 in start_thread (arg=<optimized out>)
    at pthread_create.c:463
#12 0x00007fda9b7bbe6f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
3:
{noformat}
#0  0x00007f761a027858 in EVP_PKEY_up_ref (pkey=0x1a0bf8b480166)
    at crypto/evp/p_lib.c:160
#1  0x00007f761a3898a4 in ssl_cert_dup (cert=0x7f761b85a6a0 <pn_link_incref>)
    at ssl/ssl_cert.c:99
#2  0x00007f761a394b98 in SSL_new (ctx=0x7f761ba77940 <clazz>)
    at ssl/ssl_lib.c:716
#3  0x00007f761b8680c8 in init_ssl_socket (transport=0xcbede0, ssl=0xc92d70)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#4  0x00007f761b869464 in init_ssl_socket (ssl=<optimized out>,
    transport=<optimized out>)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#5  pn_ssl_init (ssl0=<optimized out>, domain=<optimized out>,
    session_id=session_id@entry=0x0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:822
#6  0x00007f761bab09ef in qdr_handle_authentication_service_connection_event (
    e=e@entry=0xc78900) at /build/qpid-dispatch-src/src/remote_sasl.c:623
#7  0x00007f761bacda2c in handle (qd_server=qd_server@entry=0x85bf60,
    e=e@entry=0xc78900, pn_conn=pn_conn@entry=0xc35ba0, ctx=ctx@entry=0x0)
    at /build/qpid-dispatch-src/src/server.c:864
#8  0x00007f761bace2e4 in thread_run (arg=arg@entry=0x85bf60)
    at /build/qpid-dispatch-src/src/server.c:973
#9  0x00007f761bace590 in qd_server_run (qd=<optimized out>)
    at /build/qpid-dispatch-src/src/server.c:1247
#10 0x000000000040182c in main_process (
    config_path=0x7ffecc430965 "/tmp/qdrouterd.conf",
    python_pkgdir=<optimized out>, fd=2)
    at /build/qpid-dispatch-src/router/src/main.c:112
#11 0x0000000000401589 in main (argc=3, argv=0x7ffecc42e848)
    at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
 4:
{noformat}
#0  __GI_abort () at abort.c:107
#1  0x00007f3308a8e7b7 in __libc_message (action=action@entry=do_abort,
    fmt=fmt@entry=0x7f3308b98359 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#2  0x00007f3308a94c4c in malloc_printerr (
    str=str@entry=0x7f3308b99c40 "munmap_chunk(): invalid pointer")
    at malloc.c:5350
#3  0x00007f3308a951a4 in munmap_chunk (p=<optimized out>) at malloc.c:2846
#4  0x00007f33084d39f2 in OPENSSL_sk_free (st=0x7f32f00cc910)
    at crypto/stack/stack.c:273
#5  0x00007f33084718e4 in sk_void_free (sk=<optimized out>)
    at include/openssl/crypto.h:94
#6  CRYPTO_free_ex_data (class_index=class_index@entry=1,
    obj=obj@entry=0x1bcd2c0, ad=ad@entry=0x1bcd390) at crypto/ex_data.c:361
#7  0x00007f33087d84be in SSL_CTX_free (a=0x1bcd2c0) at ssl/ssl_lib.c:2718
#8  0x00007f3309caecb7 in pn_ssl_domain_free (domain=0x1a240d0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:558
#9  0x00007f3309caf6b7 in pn_ssl_free (
    transport=transport@entry=0x7f32f02940e0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:887
#10 0x00007f3309ca5108 in pn_transport_finalize (object=0x7f32f02940e0)
    at /build/qpid-proton-src/c/src/core/transport.c:663
#11 0x00007f3309c971f9 in pn_class_decref (clazz=0x7f3309ebdcc0 <clazz>,
    clazz@entry=0x7f3309ebd540 <PN_OBJECT>, object=0x7f32f02940e0)
    at /build/qpid-proton-src/c/src/core/object/object.c:95
#12 0x00007f3309c9740f in pn_decref (object=<optimized out>)
    at /build/qpid-proton-src/c/src/core/object/object.c:253
#13 0x00007f3309ca4dd1 in pn_transport_free (transport=<optimized out>)
    at /build/qpid-proton-src/c/src/core/transport.c:643
#14 0x00007f3309c9ff14 in pn_connection_driver_destroy (
    d=d@entry=0x7f32f0293f18)
    at /build/qpid-proton-src/c/src/core/connection_driver.c:94
#15 0x00007f3309a805f4 in pconnection_final_free (pc=pc@entry=0x7f32f0293970)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:889
#16 0x00007f3309a814ec in pconnection_cleanup (pc=pc@entry=0x7f32f0293970)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:905
#17 0x00007f3309a841d2 in pconnection_done (pc=0x7f32f0293970)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:1045
#18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f32f0293f10)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:2167
#19 0x00007f3309f1431c in thread_run (arg=0x16a4f60)
    at /build/qpid-dispatch-src/src/server.c:990
#20 0x00007f3309865594 in start_thread (arg=<optimized out>)
    at pthread_create.c:463
#21 0x00007f3308b0ee6f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
5:
{noformat}
#0  0x00007fd87854eae3 in SSL_new (ctx=0x712f646c6975622f) at ssl/ssl_lib.c:677
#1  0x00007fd879a220c8 in init_ssl_socket (transport=0x7fd8600eb4e0,
    ssl=0x7fd85c0ec390) at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#2  0x00007fd879a23464 in init_ssl_socket (ssl=<optimized out>,
    transport=<optimized out>)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#3  pn_ssl_init (ssl0=<optimized out>, domain=<optimized out>,
    session_id=session_id@entry=0x0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:822
#4  0x00007fd879c6a9ef in qdr_handle_authentication_service_connection_event (
    e=e@entry=0x7fd85c055040) at /build/qpid-dispatch-src/src/remote_sasl.c:623
#5  0x00007fd879c87a2c in handle (qd_server=qd_server@entry=0xc9ef60,
    e=e@entry=0x7fd85c055040, pn_conn=pn_conn@entry=0x7fd860121790,
    ctx=ctx@entry=0x0) at /build/qpid-dispatch-src/src/server.c:864
#6  0x00007fd879c882e4 in thread_run (arg=0xc9ef60)
    at /build/qpid-dispatch-src/src/server.c:973
#7  0x00007fd8795d9594 in start_thread (arg=<optimized out>)
    at pthread_create.c:463
#8  0x00007fd878882e6f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
 


was (Author: calohmn):
Here are backtraces from separate core dumps:

1:
{noformat}
#0  0x00007f4a0578f4fd in pn_list_get (list=0x1424900, index=24140)
    at /build/qpid-proton-src/c/src/core/object/list.c:42
#1  0x00007f4a0579866f in pni_session_bound (ssn=<optimized out>)
    at /build/qpid-proton-src/c/src/core/engine.c:1021
#2  0x00007f4a0579892b in pn_connection_bound (
    connection=connection@entry=0x14244a0)
    at /build/qpid-proton-src/c/src/core/engine.c:157
#3  0x00007f4a0579e268 in pn_transport_bind (transport=0xf1f760,
    connection=0x14244a0) at /build/qpid-proton-src/c/src/core/transport.c:706
#4  0x00007f4a05797d35 in batch_next (batch=0x1461330)
    at /build/qpid-proton-src/c/src/core/connection_driver.c:41
#5  0x00007f4a0557a6f1 in pconnection_batch_next ()
    at /build/qpid-proton-src/c/src/proactor/epoll.c:948
#6  0x00007f4a05a0c2fb in thread_run (arg=arg@entry=0xf0bf60)
    at /build/qpid-dispatch-src/src/server.c:976
#7  0x00007f4a05a0c590 in qd_server_run (qd=<optimized out>)
    at /build/qpid-dispatch-src/src/server.c:1247
#8  0x000000000040182c in main_process (
    config_path=0x7fff809c4965 "/tmp/qdrouterd.conf",
    python_pkgdir=<optimized out>, fd=2)
    at /build/qpid-dispatch-src/router/src/main.c:112
#9  0x0000000000401589 in main (argc=3, argv=0x7fff809c3ba8)
    at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
2:
{noformat}
#0  ssl_cert_dup (cert=0x0) at ssl/ssl_cert.c:89
#1  0x00007fda9b487b98 in SSL_new (ctx=0x1fb8a40) at ssl/ssl_lib.c:716
#2  0x00007fda9c95b0c8 in init_ssl_socket (transport=0x7fda8014ec60,
    ssl=ssl@entry=0x7fda8004dc70)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#3  0x00007fda9c95bc18 in init_ssl_socket (ssl=0x7fda8004dc70,
    transport=0x7fda8014ec60)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#4  process_input_ssl (transport=0x7fda8014ec60, layer=0,
    input_data=0x7fda80163f90 "\220\a", available=0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:963
#5  0x00007fda9c95317a in transport_consume (
    transport=transport@entry=0x7fda8014ec60)
    at /build/qpid-proton-src/c/src/core/transport.c:1821
#6  0x00007fda9c954eda in pn_transport_close_tail (transport=0x7fda8014ec60)
    at /build/qpid-proton-src/c/src/core/transport.c:2972
#7  0x00007fda9c94cdea in pn_connection_driver_read_close (
    d=d@entry=0x7fda800e8968)
    at /build/qpid-proton-src/c/src/core/connection_driver.c:114
#8  0x00007fda9c72f5f8 in pconnection_process (pc=pc@entry=0x7fda800e83c0,
    events=events@entry=0, timeout=timeout@entry=false,
    topup=topup@entry=true, is_io_2=is_io_2@entry=false)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:1230
#9  0x00007fda9c72f754 in pconnection_batch_next ()
    at /build/qpid-proton-src/c/src/proactor/epoll.c:953
#10 0x00007fda9cbc12fb in thread_run (arg=0x1bd0f60)
    at /build/qpid-dispatch-src/src/server.c:976
#11 0x00007fda9c512594 in start_thread (arg=<optimized out>)
    at pthread_create.c:463
#12 0x00007fda9b7bbe6f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}
3:
{noformat}
#0  0x00007f761a027858 in EVP_PKEY_up_ref (pkey=0x1a0bf8b480166)
    at crypto/evp/p_lib.c:160
#1  0x00007f761a3898a4 in ssl_cert_dup (cert=0x7f761b85a6a0 <pn_link_incref>)
    at ssl/ssl_cert.c:99
#2  0x00007f761a394b98 in SSL_new (ctx=0x7f761ba77940 <clazz>)
    at ssl/ssl_lib.c:716
#3  0x00007f761b8680c8 in init_ssl_socket (transport=0xcbede0, ssl=0xc92d70)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1235
#4  0x00007f761b869464 in init_ssl_socket (ssl=<optimized out>,
    transport=<optimized out>)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:1232
#5  pn_ssl_init (ssl0=<optimized out>, domain=<optimized out>,
    session_id=session_id@entry=0x0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:822
#6  0x00007f761bab09ef in qdr_handle_authentication_service_connection_event (
    e=e@entry=0xc78900) at /build/qpid-dispatch-src/src/remote_sasl.c:623
#7  0x00007f761bacda2c in handle (qd_server=qd_server@entry=0x85bf60,
    e=e@entry=0xc78900, pn_conn=pn_conn@entry=0xc35ba0, ctx=ctx@entry=0x0)
    at /build/qpid-dispatch-src/src/server.c:864
#8  0x00007f761bace2e4 in thread_run (arg=arg@entry=0x85bf60)
    at /build/qpid-dispatch-src/src/server.c:973
#9  0x00007f761bace590 in qd_server_run (qd=<optimized out>)
    at /build/qpid-dispatch-src/src/server.c:1247
#10 0x000000000040182c in main_process (
    config_path=0x7ffecc430965 "/tmp/qdrouterd.conf",
    python_pkgdir=<optimized out>, fd=2)
    at /build/qpid-dispatch-src/router/src/main.c:112
#11 0x0000000000401589 in main (argc=3, argv=0x7ffecc42e848)
    at /build/qpid-dispatch-src/router/src/main.c:360
{noformat}
 4:
{noformat}
#0  __GI_abort () at abort.c:107
#1  0x00007f3308a8e7b7 in __libc_message (action=action@entry=do_abort,
    fmt=fmt@entry=0x7f3308b98359 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#2  0x00007f3308a94c4c in malloc_printerr (
    str=str@entry=0x7f3308b99c40 "munmap_chunk(): invalid pointer")
    at malloc.c:5350
#3  0x00007f3308a951a4 in munmap_chunk (p=<optimized out>) at malloc.c:2846
#4  0x00007f33084d39f2 in OPENSSL_sk_free (st=0x7f32f00cc910)
    at crypto/stack/stack.c:273
#5  0x00007f33084718e4 in sk_void_free (sk=<optimized out>)
    at include/openssl/crypto.h:94
#6  CRYPTO_free_ex_data (class_index=class_index@entry=1,
    obj=obj@entry=0x1bcd2c0, ad=ad@entry=0x1bcd390) at crypto/ex_data.c:361
#7  0x00007f33087d84be in SSL_CTX_free (a=0x1bcd2c0) at ssl/ssl_lib.c:2718
#8  0x00007f3309caecb7 in pn_ssl_domain_free (domain=0x1a240d0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:558
#9  0x00007f3309caf6b7 in pn_ssl_free (
    transport=transport@entry=0x7f32f02940e0)
    at /build/qpid-proton-src/c/src/ssl/openssl.c:887
#10 0x00007f3309ca5108 in pn_transport_finalize (object=0x7f32f02940e0)
    at /build/qpid-proton-src/c/src/core/transport.c:663
#11 0x00007f3309c971f9 in pn_class_decref (clazz=0x7f3309ebdcc0 <clazz>,
    clazz@entry=0x7f3309ebd540 <PN_OBJECT>, object=0x7f32f02940e0)
    at /build/qpid-proton-src/c/src/core/object/object.c:95
#12 0x00007f3309c9740f in pn_decref (object=<optimized out>)
    at /build/qpid-proton-src/c/src/core/object/object.c:253
#13 0x00007f3309ca4dd1 in pn_transport_free (transport=<optimized out>)
    at /build/qpid-proton-src/c/src/core/transport.c:643
#14 0x00007f3309c9ff14 in pn_connection_driver_destroy (
    d=d@entry=0x7f32f0293f18)
    at /build/qpid-proton-src/c/src/core/connection_driver.c:94
#15 0x00007f3309a805f4 in pconnection_final_free (pc=pc@entry=0x7f32f0293970)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:889
#16 0x00007f3309a814ec in pconnection_cleanup (pc=pc@entry=0x7f32f0293970)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:905
#17 0x00007f3309a841d2 in pconnection_done (pc=0x7f32f0293970)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:1045
#18 pn_proactor_done (p=<optimized out>, batch=batch@entry=0x7f32f0293f10)
    at /build/qpid-proton-src/c/src/proactor/epoll.c:2167
#19 0x00007f3309f1431c in thread_run (arg=0x16a4f60)
    at /build/qpid-dispatch-src/src/server.c:990
#20 0x00007f3309865594 in start_thread (arg=<optimized out>)
    at pthread_create.c:463
#21 0x00007f3308b0ee6f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
{noformat}

> Dispatch Router sporadically goes into a state where TLS connections to the auth service
fail
> ---------------------------------------------------------------------------------------------
>
>                 Key: DISPATCH-1086
>                 URL: https://issues.apache.org/jira/browse/DISPATCH-1086
>             Project: Qpid Dispatch
>          Issue Type: Bug
>    Affects Versions: 1.1.0
>            Reporter: Keith Wall
>            Priority: Major
>
> Whilst running performance tests against Enmasse, we periodically see a problem where
Dispatch Router (1.1.0) goes into a state where fails to form TLS connections to the authservice.
When this occurs, the router needs to be restarted to restore service. There does not seem
to be a pattern to when the issue occurs, but in all cases where it has been seen, the test
case included tens or hundreds of concurrently formed connections.
> The following message is written to the log:
>  
> {noformat}
> 2018-07-06 10:38:45.543519 +0000 AUTHSERVICE (warning) Cannot initialise SSL{noformat}
>  Unfortunately turning up the router logging (using the following command) reveal no
more useful information. This Proton improvement JIRA was raised to include the diagnostics
from OpenSSL.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message