storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kim, Peter" <Peter....@blackrock.com>
Subject Storm 2.0.0 Kerberos Socket Ouput
Date Thu, 22 Oct 2020 13:59:35 GMT
Hello,

Recently, we’ve implemented storm 2.0.0 with Kerberos. Before integrating Kerberos, the
logs were fine and no mention of socket output being closed but since we implemented Kerberos,
we’re seeing the following in our logs :

2020-10-21 14:39:53.529 o.a.s.t.t.TIOStreamTransport refresh-connections-timer [WARN] Error
closing output stream.
java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116) ~[?:1.8.0_45]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_45]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_45]
        at org.apache.storm.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
[storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.thrift.transport.TSocket.close(TSocket.java:235) [storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:106) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.<init>(SupervisorClient.java:45)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClientAs(SupervisorClient.java:73)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClient(SupervisorClient.java:61)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.WorkerState.getLocalAssignment(WorkerState.java:647)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.WorkerState.refreshConnections(WorkerState.java:367)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.WorkerState.refreshConnections(WorkerState.java:356)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker$$Lambda$61/1922191129.run(Unknown Source)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:110) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226) [storm-client-2.0.0.jar:2.0.0]
2020-10-21 14:43:37.827 o.a.s.t.t.TIOStreamTransport heartbeat-timer [WARN] Error closing
output stream.
java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116) ~[?:1.8.0_45]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_45]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_45]
        at org.apache.storm.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
[storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.thrift.transport.TSocket.close(TSocket.java:235) [storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:106) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.<init>(SupervisorClient.java:45)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClientAs(SupervisorClient.java:73)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClient(SupervisorClient.java:61)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker.heartbeatToMasterIfLocalbeatFail(Worker.java:429)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker.doHeartBeat(Worker.java:344) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker.lambda$loadWorker$1(Worker.java:189) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker$$Lambda$30/435297131.run(Unknown Source)
[storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:110) [storm-client-2.0.0.jar:2.0.0]
       at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226) [storm-client-2.0.0.jar:2.0.0]
2020-10-21 14:43:37.827 o.a.s.d.w.Worker heartbeat-timer [WARN] Exception when send heartbeat
to local supervisor

I’ve done some searching and tried adjusting the worker heartbeat frequency sec but that
did more harm than good. I’ve followed the trace and the only configuration that I see that
is being pulled in is the worker.heartbeat.freqs.secs set at 1.

Is this a known issue or has anyone come across this before? Any thoughts on what might be
the issue and if it’s critical? Thanks for your help.

Best,
Peter


This message may contain information that is confidential or privileged. If you are not the
intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers
for further information.  Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy
for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.

© 2020 BlackRock, Inc. All rights reserved.

Mime
View raw message