spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Femi Anthony <femib...@gmail.com>
Subject Re: Insufficient memory for Java Runtime
Date Wed, 14 Mar 2018 10:06:33 GMT
Try specifying executor memory.

On Tue, Mar 13, 2018 at 5:15 PM, Shiyuan <gshy2014@gmail.com> wrote:

> Hi Spark-Users,
>   I encountered the problem of "insufficient memory". The error is logged
> in the file with a name " hs_err_pid86252.log"(attached in the end of this
> email).
>
> I launched the spark job by " spark-submit --driver-memory 40g --master
> yarn --deploy-mode client".  The spark session was created with 10
> executors each with 60g memory. The data access pattern is pretty simple, I
> keep reading some spark dataframe from hdfs one by one, filter, join with
> another dataframe,  and then append the results to an dataframe:
> for i= 1,2,3....
>     df1 = spark.read.parquet(file_i)
>     df_r = df1.filter(...). join(df2)
>     df_all = df_all.union(df_r)
>
> each file_i is quite small, only a few GB, but there are a lot of such
> files. after filtering and join, each df_r is also quite small. When the
> program failed, df_all had only 10k rows which should be around 10GB.  Each
> machine in the cluster has round 80GB memory and 1TB disk space and  only
> one user was using the cluster when it failed due to insufficient memory.
> My questions are:
> i).  The log file showed that it failed to allocate 8G committing memory.
> But how could that happen when the driver and executors have more than 40g
> free memory. In fact, only transformations but no actions had run when the
> program failed.  As I understand, only DAG and book-keeping work is done
> during dataframe transformation, no data is brought into the memory.  Why
> spark still tries to allocate such large memory?
> ii). Could manually running garbage collection help?
> iii). Did I mis-specify some runtime parameter for jvm, yarn, or spark?
>
>
> Any help or references are appreciated!
>
> The content of hs_err_pid86252,log:
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
> # Native memory allocation (mmap) failed to map 8663334912
> <(866)%20333-4912> bytes(~8G) for committing reserved memory.
> # Possible reasons:
> #   The system is out of physical RAM or swap space
> #   In 32 bit mode, the process size limit was hit
> # Possible solutions:
> #   Reduce memory load on the system
> #   Increase physical memory or swap space
> #   Check if swap backing store is full
> #   Use 64 bit Java on a 64 bit OS
> #   Decrease Java heap size (-Xmx/-Xms)
> #   Decrease number of Java threads
> #   Decrease Java thread stack sizes (-Xss)
> #   Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> #  Out of Memory Error (os_linux.cpp:2643), pid=86252,
> tid=0x00007fd69e683700
> #
> # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build
> 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
> # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 )
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
>
> ---------------  T H R E A D  ---------------
>
> Current thread (0x00007fe0bc08c000):  VMThread [stack: 0x00007fd69e583000,0x00007fd69e684000]
> [id=86295]
>
>


-- 
http://www.femibyte.com/twiki5/bin/view/Tech/
http://www.nextmatrix.com
"Great spirits have always encountered violent opposition from mediocre
minds." - Albert Einstein.

Mime
View raw message