spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject BlockManager and ShuffleManager = can getLocalBytes be ever used for shuffle blocks?
Date Thu, 16 Apr 2020 11:25:49 GMT
Hi,

While trying to understand the relationship of BlockManager
and ShuffleManager I found that ShuffleManager is used for shuffle block
data [1] (and that makes sense).

What I found quite surprising is that BlockManager can call getLocalBytes
for non-shuffle blocks that in turn does...fetching shuffle blocks (!) [2].

That begs the question, who could want this? In other words, what other
components would want to call BlockManager.getLocalBytes for shuffle
blocks? The only caller is TorrentBroadcast which is not for shuffle
blocks, is it?

In other words, getLocalBytes should NOT be bothering itself with fetching
local shuffle blocks as that's a responsibility of someone else,
i.e. ShuffleManager.

Is this correct?

I'd appreciate any help you can provide so I can understand the Spark core
better. Thanks!

[1]
https://github.com/apache/spark/blob/31734399d57f3c128e66b0f97ef83eb4c9165978/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L382
[2]
https://github.com/apache/spark/blob/31734399d57f3c128e66b0f97ef83eb4c9165978/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L637

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski

<https://twitter.com/jaceklaskowski>

Mime
View raw message