trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Peach <jpe...@apache.org>
Subject Re: Traffic Server 4.0.1 Segmentation fault
Date Fri, 04 Oct 2013 15:17:58 GMT
On Oct 3, 2013, at 10:52 PM, Vladyslav Bachynskyi <v.bachynskyi@gmail.com> wrote:

> Hi All,
> 
> One of my ats servers was crashed :(
> My system information:
> 
> Centos 6.4 - 2.6.32-358.18.1.el6.x86_64
> Apache Traffic Server - traffic_server - 4.0.1 - (build # 8913 on Sep  9
> 2013 at 13:16:57)
> 
> traffic.out output:
> 
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /opt/trafficserver-4.0.1/bin/traffic_server - STACK TRACE:
> /lib64/libpthread.so.0(+0x3b3300f500)[0x2abf3e03a500]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN10HttpTunnel12producer_runEP18HttpTunnelProducer+0x25f)[0x56b8ef]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN10HttpTunnel10tunnel_runEP18HttpTunnelProducer+0xd9)[0x56ca49]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM23setup_internal_transferEMS_FiiPvE+0x633)[0x534ce3]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x3e1)[0x5353c1]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x547)[0x532e87]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x3b7)[0x52b637]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x8b)[0x5314fb]
> /opt/trafficserver-4.0.1/bin/traffic_server(TSHttpTxnReenable+0x404)[0x4b8694]
> /usr/local/lib/libatscppapi.so.1(_ZN9atscppapi5utils8internal20invokePluginForEventEPNS_17TransactionPluginEP13tsapi_httptxn7TSEvent+0x195)[0x2ac01c4ded45]

It looks like you are using the LinkedIn C++ API in a transform plugin? Brian, does this ring
any bells?

> /usr/local/lib/libatscppapi.so.1(+0xfa8e)[0x2ac01c4e0a8e]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x116)[0x52b396]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x114)[0x532a54]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x527a0d]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x531308]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d6)[0x4e7d46]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a2fcf]
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN7EThread7executeEv+0x683)[0x6a3b93]
> /opt/trafficserver-4.0.1/bin/traffic_server[0x6a1e6a]
> /lib64/libpthread.so.0(+0x3b33007851)[0x2abf3e032851]
> /lib64/libc.so.6(clone+0x6d)[0x3b32ce894d]
> 
> Linux syslog output (it's not a full log output, as messages repeats. I
> can provide full log output if needed):
> 
> Oct  3 23:01:53 ats-node01 kernel: [ET_NET 22][2719]: segfault at 10 ip
> 000000000056b8ef sp 00002abf459156e0 error 4 in
> traffic_server[400000+350000]
> Oct  3 23:01:53 ats-node01 abrt[9515]:
> /opt/trafficserver-4.0.1/core.2696 fd(-1) is not a regular file with
> link count 1: Permission denied
> Oct  3 23:04:22 ats-node01 kernel: INFO: task [ET_NET 0]:2696 blocked
> for more than 120 seconds.
> Oct  3 23:04:22 ats-node01 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Oct  3 23:04:22 ats-node01 kernel: [ET_NET 0]    D 0000000000000009    
> 0  2696   2648 0x00000080
> Oct  3 23:04:22 ats-node01 kernel: ffff8804ba037c98 0000000000000082
> ffff8804ba037c60 ffff8804ba037c5c
> Oct  3 23:04:22 ats-node01 kernel: ffff8804ba037c68 ffff88033fc25400
> ffff880028256700 00000000000009fd
> Oct  3 23:04:22 ats-node01 kernel: ffff8804bb8c45f8 ffff8804ba037fd8
> 000000000000fb88 ffff8804bb8c45f8
> Oct  3 23:04:22 ats-node01 kernel: Call Trace:
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073095>] exit_mm+0x95/0x180
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff810734df>] do_exit+0x15f/0x870
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8150e130>] ?
> thread_return+0x4e/0x76e
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8109b2c1>] ?
> lock_hrtimer_base+0x31/0x60
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073c48>]
> do_group_exit+0x58/0xd0
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81088f36>]
> get_signal_to_deliver+0x1f6/0x460
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff811c7ac6>] ?
> ep_poll+0x306/0x330
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81063410>] ?
> default_wake_function+0x0/0x20
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8121bed6>] ?
> security_file_permission+0x16/0x20
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100aa80>]
> do_notify_resume+0x90/0xc0
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
> Oct  3 23:04:22 ats-node01 kernel: INFO: task traffic_server:2697
> blocked for more than 120 seconds.
> Oct  3 23:04:22 ats-node01 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Oct  3 23:04:22 ats-node01 kernel: traffic_serve D 0000000000000000    
> 0  2697   2648 0x00000080
> Oct  3 23:04:22 ats-node01 kernel: ffff8804bb8d9c98 0000000000000082
> ffff8804bb8d9cb8 ffffffff8112f9bc
> Oct  3 23:04:22 ats-node01 kernel: ffff8804bb8d9c68 ffffffff810aa5e0
> ffff8804bb8d9c28 ffffffff81277679
> Oct  3 23:04:22 ats-node01 kernel: ffff8804ba4a6638 ffff8804bb8d9fd8
> 000000000000fb88 ffff8804ba4a6638
> Oct  3 23:04:22 ats-node01 kernel: Call Trace:
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8112f9bc>] ?
> release_pages+0x21c/0x250
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff810aa5e0>] ?
> exit_robust_list+0x90/0x160
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81277679>] ?
> cpumask_next_and+0x29/0x50
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073095>] exit_mm+0x95/0x180
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff810734df>] do_exit+0x15f/0x870
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073c48>]
> do_group_exit+0x58/0xd0
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81088f36>]
> get_signal_to_deliver+0x1f6/0x460
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8109c0f7>] ?
> hrtimer_nanosleep+0xe7/0x180
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8109af20>] ?
> hrtimer_wakeup+0x0/0x30
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100aa80>]
> do_notify_resume+0x90/0xc0
> Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
> 
> =========== [ SKIP ] ===============
> 
> Oct  3 23:04:47 ats-node01 abrt[9515]: Saved core dump of pid 2696
> (/opt/trafficserver-4.0.1/bin/traffic_server) to
> /var/spool/abrt/ccpp-2013-10-03-23:01:53-2696 (12643663872 bytes)
> Oct  3 23:04:47 ats-node01 abrtd: Directory
> 'ccpp-2013-10-03-23:01:53-2696' creation detected
> Oct  3 23:04:47 ats-node01 abrtd: Executable
> '/opt/trafficserver-4.0.1/bin/traffic_server' doesn't belong to any package
> Oct  3 23:04:47 ats-node01 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2013-10-03-23:01:53-2696' exited with 1
> Oct  3 23:04:47 ats-node01 abrtd: Corrupted or bad directory
> '/var/spool/abrt/ccpp-2013-10-03-23:01:53-2696', deleting
> Oct  3 23:04:48 ats-node01 traffic_cop[2639]: (test) read failed [104
> 'Connection reset by peer']
> Oct  3 23:04:48 ats-node01 traffic_cop[2639]: server heartbeat failed [1]
> Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
> Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> FATAL:  (last system error 104: Connection reset by peer)
> Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
> Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> ERROR:  (last system error 32: Broken pipe)
> Oct  3 23:04:48 ats-node01 traffic_cop[2639]: cop received child status
> signal [2648 256]
> Oct  3 23:04:48 ats-node01 traffic_cop[2639]: traffic_manager not
> running, making sure traffic_server is dead
> Oct  3 23:04:48 ats-node01 traffic_cop[2639]: spawning traffic_manager
> Oct  3 23:04:48 ats-node01 traffic_manager[9533]: NOTE: --- Manager
> Starting ---
> Oct  3 23:04:48 ats-node01 traffic_manager[9533]: NOTE: Manager Version:
> Apache Traffic Server - traffic_manager - 4.0.1 - (build # 8913 on Sep 
> 9 2013 at 13:15:00)
> Oct  3 23:04:48 ats-node01 traffic_manager[9533]: NOTE:
> RLIMIT_NOFILE(7):cur(30000),max(30000)
> Oct  3 23:04:50 ats-node01 traffic_server[9578]: NOTE: --- Server
> Starting ---
> Oct  3 23:04:50 ats-node01 traffic_server[9578]: NOTE: Server Version:
> Apache Traffic Server - traffic_server - 4.0.1 - (build # 8913 on Sep  9
> 2013 at 13:16:57)
> Oct  3 23:04:58 ats-node01 traffic_cop[2639]: server heartbeat succeeded
> 
> 
> 
> Any suggestions will be very appreciated!


Mime
View raw message