trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Geffon <bri...@apache.org>
Subject Re: Traffic Server 4.0.1 Segmentation fault
Date Fri, 04 Oct 2013 21:28:05 GMT
I haven't seen anything, but it also looks like you're using a
Transformation Plugin, if you'd be willing to share code we could possibly
help.

Brian


On Fri, Oct 4, 2013 at 8:17 AM, James Peach <jpeach@apache.org> wrote:

> On Oct 3, 2013, at 10:52 PM, Vladyslav Bachynskyi <v.bachynskyi@gmail.com>
> wrote:
>
> > Hi All,
> >
> > One of my ats servers was crashed :(
> > My system information:
> >
> > Centos 6.4 - 2.6.32-358.18.1.el6.x86_64
> > Apache Traffic Server - traffic_server - 4.0.1 - (build # 8913 on Sep  9
> > 2013 at 13:16:57)
> >
> > traffic.out output:
> >
> > NOTE: Traffic Server received Sig 11: Segmentation fault
> > /opt/trafficserver-4.0.1/bin/traffic_server - STACK TRACE:
> > /lib64/libpthread.so.0(+0x3b3300f500)[0x2abf3e03a500]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN10HttpTunnel12producer_runEP18HttpTunnelProducer+0x25f)[0x56b8ef]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN10HttpTunnel10tunnel_runEP18HttpTunnelProducer+0xd9)[0x56ca49]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM23setup_internal_transferEMS_FiiPvE+0x633)[0x534ce3]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x3e1)[0x5353c1]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x547)[0x532e87]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x3b7)[0x52b637]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x8b)[0x5314fb]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(TSHttpTxnReenable+0x404)[0x4b8694]
> >
> /usr/local/lib/libatscppapi.so.1(_ZN9atscppapi5utils8internal20invokePluginForEventEPNS_17TransactionPluginEP13tsapi_httptxn7TSEvent+0x195)[0x2ac01c4ded45]
>
> It looks like you are using the LinkedIn C++ API in a transform plugin?
> Brian, does this ring any bells?
>
> > /usr/local/lib/libatscppapi.so.1(+0xfa8e)[0x2ac01c4e0a8e]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x116)[0x52b396]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x114)[0x532a54]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x527a0d]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x531308]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d6)[0x4e7d46]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a2fcf]
> >
> /opt/trafficserver-4.0.1/bin/traffic_server(_ZN7EThread7executeEv+0x683)[0x6a3b93]
> > /opt/trafficserver-4.0.1/bin/traffic_server[0x6a1e6a]
> > /lib64/libpthread.so.0(+0x3b33007851)[0x2abf3e032851]
> > /lib64/libc.so.6(clone+0x6d)[0x3b32ce894d]
> >
> > Linux syslog output (it's not a full log output, as messages repeats. I
> > can provide full log output if needed):
> >
> > Oct  3 23:01:53 ats-node01 kernel: [ET_NET 22][2719]: segfault at 10 ip
> > 000000000056b8ef sp 00002abf459156e0 error 4 in
> > traffic_server[400000+350000]
> > Oct  3 23:01:53 ats-node01 abrt[9515]:
> > /opt/trafficserver-4.0.1/core.2696 fd(-1) is not a regular file with
> > link count 1: Permission denied
> > Oct  3 23:04:22 ats-node01 kernel: INFO: task [ET_NET 0]:2696 blocked
> > for more than 120 seconds.
> > Oct  3 23:04:22 ats-node01 kernel: "echo 0 >
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Oct  3 23:04:22 ats-node01 kernel: [ET_NET 0]    D 0000000000000009
> > 0  2696   2648 0x00000080
> > Oct  3 23:04:22 ats-node01 kernel: ffff8804ba037c98 0000000000000082
> > ffff8804ba037c60 ffff8804ba037c5c
> > Oct  3 23:04:22 ats-node01 kernel: ffff8804ba037c68 ffff88033fc25400
> > ffff880028256700 00000000000009fd
> > Oct  3 23:04:22 ats-node01 kernel: ffff8804bb8c45f8 ffff8804ba037fd8
> > 000000000000fb88 ffff8804bb8c45f8
> > Oct  3 23:04:22 ats-node01 kernel: Call Trace:
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073095>]
> exit_mm+0x95/0x180
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff810734df>]
> do_exit+0x15f/0x870
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8150e130>] ?
> > thread_return+0x4e/0x76e
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8109b2c1>] ?
> > lock_hrtimer_base+0x31/0x60
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073c48>]
> > do_group_exit+0x58/0xd0
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81088f36>]
> > get_signal_to_deliver+0x1f6/0x460
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100a265>]
> do_signal+0x75/0x800
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff811c7ac6>] ?
> > ep_poll+0x306/0x330
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81063410>] ?
> > default_wake_function+0x0/0x20
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8121bed6>] ?
> > security_file_permission+0x16/0x20
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100aa80>]
> > do_notify_resume+0x90/0xc0
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100b341>]
> int_signal+0x12/0x17
> > Oct  3 23:04:22 ats-node01 kernel: INFO: task traffic_server:2697
> > blocked for more than 120 seconds.
> > Oct  3 23:04:22 ats-node01 kernel: "echo 0 >
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Oct  3 23:04:22 ats-node01 kernel: traffic_serve D 0000000000000000
> > 0  2697   2648 0x00000080
> > Oct  3 23:04:22 ats-node01 kernel: ffff8804bb8d9c98 0000000000000082
> > ffff8804bb8d9cb8 ffffffff8112f9bc
> > Oct  3 23:04:22 ats-node01 kernel: ffff8804bb8d9c68 ffffffff810aa5e0
> > ffff8804bb8d9c28 ffffffff81277679
> > Oct  3 23:04:22 ats-node01 kernel: ffff8804ba4a6638 ffff8804bb8d9fd8
> > 000000000000fb88 ffff8804ba4a6638
> > Oct  3 23:04:22 ats-node01 kernel: Call Trace:
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8112f9bc>] ?
> > release_pages+0x21c/0x250
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff810aa5e0>] ?
> > exit_robust_list+0x90/0x160
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81277679>] ?
> > cpumask_next_and+0x29/0x50
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073095>]
> exit_mm+0x95/0x180
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff810734df>]
> do_exit+0x15f/0x870
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81073c48>]
> > do_group_exit+0x58/0xd0
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff81088f36>]
> > get_signal_to_deliver+0x1f6/0x460
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100a265>]
> do_signal+0x75/0x800
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8109c0f7>] ?
> > hrtimer_nanosleep+0xe7/0x180
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8109af20>] ?
> > hrtimer_wakeup+0x0/0x30
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100aa80>]
> > do_notify_resume+0x90/0xc0
> > Oct  3 23:04:22 ats-node01 kernel: [<ffffffff8100b341>]
> int_signal+0x12/0x17
> >
> > =========== [ SKIP ] ===============
> >
> > Oct  3 23:04:47 ats-node01 abrt[9515]: Saved core dump of pid 2696
> > (/opt/trafficserver-4.0.1/bin/traffic_server) to
> > /var/spool/abrt/ccpp-2013-10-03-23:01:53-2696 (12643663872 bytes)
> > Oct  3 23:04:47 ats-node01 abrtd: Directory
> > 'ccpp-2013-10-03-23:01:53-2696' creation detected
> > Oct  3 23:04:47 ats-node01 abrtd: Executable
> > '/opt/trafficserver-4.0.1/bin/traffic_server' doesn't belong to any
> package
> > Oct  3 23:04:47 ats-node01 abrtd: 'post-create' on
> > '/var/spool/abrt/ccpp-2013-10-03-23:01:53-2696' exited with 1
> > Oct  3 23:04:47 ats-node01 abrtd: Corrupted or bad directory
> > '/var/spool/abrt/ccpp-2013-10-03-23:01:53-2696', deleting
> > Oct  3 23:04:48 ats-node01 traffic_cop[2639]: (test) read failed [104
> > 'Connection reset by peer']
> > Oct  3 23:04:48 ats-node01 traffic_cop[2639]: server heartbeat failed [1]
> > Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> > FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
> > Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> > FATAL:  (last system error 104: Connection reset by peer)
> > Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> > ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
> > Oct  3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0}
> > ERROR:  (last system error 32: Broken pipe)
> > Oct  3 23:04:48 ats-node01 traffic_cop[2639]: cop received child status
> > signal [2648 256]
> > Oct  3 23:04:48 ats-node01 traffic_cop[2639]: traffic_manager not
> > running, making sure traffic_server is dead
> > Oct  3 23:04:48 ats-node01 traffic_cop[2639]: spawning traffic_manager
> > Oct  3 23:04:48 ats-node01 traffic_manager[9533]: NOTE: --- Manager
> > Starting ---
> > Oct  3 23:04:48 ats-node01 traffic_manager[9533]: NOTE: Manager Version:
> > Apache Traffic Server - traffic_manager - 4.0.1 - (build # 8913 on Sep
> > 9 2013 at 13:15:00)
> > Oct  3 23:04:48 ats-node01 traffic_manager[9533]: NOTE:
> > RLIMIT_NOFILE(7):cur(30000),max(30000)
> > Oct  3 23:04:50 ats-node01 traffic_server[9578]: NOTE: --- Server
> > Starting ---
> > Oct  3 23:04:50 ats-node01 traffic_server[9578]: NOTE: Server Version:
> > Apache Traffic Server - traffic_server - 4.0.1 - (build # 8913 on Sep  9
> > 2013 at 13:16:57)
> > Oct  3 23:04:58 ats-node01 traffic_cop[2639]: server heartbeat succeeded
> >
> >
> >
> > Any suggestions will be very appreciated!
>
>

Mime
View raw message