spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HuS.Andy <hushansm...@hotmail.com>
Subject RE: Spark Executor Memory Usage
Date Mon, 01 Jun 2015 08:41:50 GMT
#1 I not sure if I got you point, as I known, Xmx is not turn into physical memory as soon
as the process running. it first loaded into virtual memory, if you heap is need more, it
will gradually increase in physical memory until to the max heap.
#2 Physical memory contains not only heap, but also stack, direct memory, shared lib, and
perm space, and also there have VSS, RSS, PSS, USS concept, you can google. 
simple says:Vss = virtual set sizeRss = resident set sizePss = proportional set size Uss =
unique set size
Best Regards,Andy Hu(胡 珊)

Date: Fri, 29 May 2015 07:41:41 -0700
Subject: Re: Spark Executor Memory Usage
From: yuzhihong@gmail.com
To: valeramoiseenko@gmail.com
CC: user@spark.apache.org

For #2, see http://unix.stackexchange.com/questions/65835/htop-reporting-much-higher-memory-usage-than-free-or-top
Cheers
On Fri, May 29, 2015 at 6:56 AM, Valerii Moisieienko <valeramoiseenko@gmail.com> wrote:
Hello!

My name is Valerii. I have noticed strange memory behaivour of Spark's

executor on my cluster. Cluster works in standalone mode with 3 workers.

Application runs in cluster mode.

From topology configuration

spark.executor.memory              1536m

I checked heap usage via JVisualVM:

http://joxi.ru/Q2KqBMdSvYpDrj

and via htop:

http://joxi.ru/Vm63RWeCvG6L2Z



I have 2 questions regarding Spark's executors memory usage:

1. Why does Max Heap Size change during executor work?

2. Why does Memory usage via htop greater than executor's heap size?



Thank you!









--

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Executor-Memory-Usage-tp23083.html

Sent from the Apache Spark User List mailing list archive at Nabble.com.



---------------------------------------------------------------------

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org

For additional commands, e-mail: user-help@spark.apache.org




 		 	   		  
Mime
View raw message