spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Larry Xiao <xia...@sjtu.edu.cn>
Subject Executor and BlockManager memory size
Date Fri, 10 Oct 2014 01:40:10 GMT
Hi all,

I'm confused about Executor and BlockManager, why they have different 
memory.
> 14/10/10 08:50:02 INFO AppClient$ClientActor: Executor added: 
> app-20141010085001-0000/2 on worker-20141010004933-brick6-35657 
> (brick6:35657) with 6 cores
> 14/10/10 08:50:02 INFO SparkDeploySchedulerBackend: Granted executor 
> ID app-20141010085001-0000/2 on hostPort brick6:35657 with 6 cores, 
> 50.0 GB RAM
>
> 14/10/10 08:50:07 INFO BlockManagerMasterActor: Registering block 
> manager brick6:53296 with 26.8 GB RAM
and on the WebUI,
> Executor ID    Address    RDD Blocks  Memory Used    Disk Used   
>  Active Tasks    Failed Tasks  Complete Tasks    Total Tasks    Task 
> Time    Input    Shuffle Read    Shuffle Write
> 0    brick3:37607    0    0.0 B / 26.8 GB    0.0 B    6    0  0    6   
>  0 ms    0.0 B    0.0 B    0.0 B
> 1    brick1:59493    0    0.0 B / 26.8 GB    0.0 B    6    0  0    6   
>  0 ms    0.0 B    0.0 B    0.0 B
> 2    brick6:53296    0    0.0 B / 26.8 GB    0.0 B    6    0  0    6   
>  0 ms    0.0 B    0.0 B    0.0 B
> 3    brick5:38543    0    0.0 B / 26.8 GB    0.0 B    6    0  0    6   
>  0 ms    0.0 B    0.0 B    0.0 B
> 4    brick2:44937    0    0.0 B / 26.8 GB    0.0 B    6    0  0    6   
>  0 ms    0.0 B    0.0 B    0.0 B
> 5    brick4:46798    0    0.0 B / 26.8 GB    0.0 B    6    0  0    6   
>  0 ms    0.0 B    0.0 B    0.0 B
> <driver>    brick0:57692    0    0.0 B / 274.6 MB    0.0 B  0    0   
>  0    0    0 ms    0.0 B    0.0 B    0.0 B
As I understand it, a worker consist of a daemon and an executor, and 
executor takes charge both execution and storage.
So does it mean that 26.8 GB is saved for storage and the rest is for 
execution?

Another question is that, throughout execution, it seems that the 
blockmanager is always almost free.
> 14/10/05 14:33:44 INFO BlockManagerInfo: Added broadcast_21_piece0 in 
> memory on brick2:57501 (size: 1669.0 B, free: 21.2 GB)
I don't know what I'm missing here.

Best regards,
Larry

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message