spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Spark driver crashed with internal error
Date Mon, 08 Apr 2019 03:00:05 GMT
Hi all,

Recently, our Spark application's (2.3.1) driver has been crashing before
exiting with the following error.

* Could not load hsdis-amd64.so; library not loadable; PrintAssembly is
disabled*
* #*
* # A fatal error has been detected by the Java Runtime Environment:*
* #*
* #  Internal Error (sharedRuntime.cpp:834), pid=40111,
tid=0x00002ac46180a700*
* #  fatal error: exception happened outside interpreter, nmethods and
vtable stubs at pc 0x00002ac1a832edb1*
* #*
* # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build
1.8.0_131-b11)*
* # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode
linux-amd64 compressed oops)*

Similar errors were reported in older JVM and Linux  kernel versions
https://bugs.openjdk.java.net/browse/JDK-8203612 but we are running on Java
1.8.0_131 and kernel 3.10.0-693.21.1.el7.x86_64.

Here is the stack in error report.

*Current thread (0x00002ac438005000):  JavaThread
"block-manager-slave-async-thread-pool-2" daemon [_thread_in_Java,
id=40362, stack(0x00002ac46170a000,0x00002ac46180b000)]*

*Stack: [0x00002ac46170a000,0x00002ac46180b000],  sp=0x00002ac461808cd0,
free space=1019k*
*Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)*
*V  [libjvm.so+0xac826a]  VMError::report_and_die()+0x2ba*
*V  [libjvm.so+0x4fd089]  report_fatal(char const*, int, char const*)+0x59*
*V  [libjvm.so+0x9c391a]
SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned
char*, SharedRuntime::ImplicitExceptionKind)+0x33a*
*V  [libjvm.so+0x92bbfa]  JVM_handle_linux_signal+0x48a*
*V  [libjvm.so+0x921e13]  signalHandler(int, siginfo*, void*)+0x43*
*C  [libpthread.so.0+0xf5d0]*
*j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5*
*j  java.lang.Thread.run()V+11*
*v  ~StubRoutines::call_stub*
*V  [libjvm.so+0x691d16]  JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056*
*V  [libjvm.so+0x692221]  JavaCalls::call_virtual(JavaValue*, KlassHandle,
Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321*
*V  [libjvm.so+0x6926c7]  JavaCalls::call_virtual(JavaValue*, Handle,
KlassHandle, Symbol*, Symbol*, Thread*)+0x47*
*V  [libjvm.so+0x72da50]  thread_entry(JavaThread*, Thread*)+0xa0*
*V  [libjvm.so+0xa76833]  JavaThread::thread_main_inner()+0x103*
*V  [libjvm.so+0xa7697c]  JavaThread::run()+0x11c*
*V  [libjvm.so+0x927568]  java_start(Thread*)+0x108*
*C  [libpthread.so.0+0x7dd5]*

Has anyone seen this kind of error before ? I could provide more
information if needed.

Thanks,
Manu Zhang

Mime
View raw message