spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manuel Sopena Ballesteros <manuel...@garvan.org.au>
Subject driver crashesneed to find out why driver keeps crashing
Date Mon, 21 Oct 2019 05:46:33 GMT
Dear Apache Spark community,

My spark driver crashes and logs does not gives enough explanation of why it happens:

INFO [2019-10-21 16:33:37,045] ({pool-6-thread-7} SchedulerFactory.java[jobStarted]:109) -
Job 20190926-163704_913596201 started by scheduler interpreter_2100843352
DEBUG [2019-10-21 16:33:37,046] ({pool-6-thread-7} RemoteInterpreterServer.java[jobRun]:632)
- Script after hooks: a = "bigword"
b = "bigword"
print(a)

for i in range(10000000):
    a += b

print(a)
__zeppelin__._displayhook()
DEBUG [2019-10-21 16:33:37,046] ({pool-6-thread-7} RemoteInterpreterEventClient.java[sendEvent]:413)
- Send Event: RemoteInterpreterEvent(type:META_INFOS, data:{"message":"Spark UI enabled","url":"http://r640-1-10-mlx.mlx:38863"})
DEBUG [2019-10-21 16:33:37,048] ({pool-5-thread-1} RemoteInterpreterEventClient.java[pollEvent]:366)
- Send event META_INFOS
DEBUG [2019-10-21 16:33:37,054] ({Thread-34} RemoteInterpreterServer.java[onUpdate]:799) -
Output Update:
DEBUG [2019-10-21 16:33:37,054] ({Thread-34} RemoteInterpreterEventClient.java[sendEvent]:413)
- Send Event: RemoteInterpreterEvent(type:OUTPUT_UPDATE, data:{"data":"","index":"0","noteId":"2ENM9X82N","paragraphId":"20190926-163704_913596201","type":"TEXT"})
DEBUG [2019-10-21 16:33:37,054] ({Thread-34} RemoteInterpreterServer.java[onAppend]:789) -
Output Append: bigword

DEBUG [2019-10-21 16:33:37,054] ({pool-5-thread-1} RemoteInterpreterEventClient.java[pollEvent]:366)
- Send event OUTPUT_UPDATE
DEBUG [2019-10-21 16:33:37,054] ({Thread-34} RemoteInterpreterEventClient.java[sendEvent]:413)
- Send Event: RemoteInterpreterEvent(type:OUTPUT_APPEND, data:{"data":"bigword\n","index":"0","noteId":"2ENM9X82N","paragraphId":"20190926-163704_913596201"})
DEBUG [2019-10-21 16:33:37,062] ({pool-5-thread-1} RemoteInterpreterEventClient.java[pollEvent]:366)
- Send event OUTPUT_APPEND
DEBUG [2019-10-21 16:33:37,145] ({pool-5-thread-3} Interpreter.java[getProperty]:222) - key:
zeppelin.spark.concurrentSQL, value: false
DEBUG [2019-10-21 16:33:37,145] ({pool-5-thread-3} Interpreter.java[getProperty]:222) - key:
zeppelin.spark.concurrentSQL, value: false
DEBUG [2019-10-21 16:33:37,225] ({pool-5-thread-3} RemoteInterpreterServer.java[resourcePoolGetAll]:1089)
- Request getAll from ZeppelinServer
ERROR [2019-10-21 16:33:40,357] ({SIGTERM handler} SignalUtils.scala[apply$mcZ$sp]:43) - RECEIVED
SIGNAL TERM
INFO [2019-10-21 16:33:40,530] ({shutdown-hook-0} Logging.scala[logInfo]:54) - Invoking stop()
from shutdown hook

I assume it is because the jvm runs out of memory but I would expect an error saying so.

Any idea?

Thank you very much
NOTICE
Please consider the environment before printing this email. This message and any attachments
are intended for the addressee named and may contain legally privileged/confidential/copyright
information. If you are not the intended recipient, you should not read, use, disclose, copy
or distribute this communication. If you have received this message in error please notify
us at once by return email and then delete both messages. We accept no liability for the distribution
of viruses or similar in electronic communications. This notice should not be removed.

Mime
View raw message