spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Masiero Vanzin (Jira)" <>
Subject [jira] [Created] (SPARK-29319) Memory and disk usage not accurate when blocks are evicted and re-loaded in memory
Date Tue, 01 Oct 2019 16:07:00 GMT
Marcelo Masiero Vanzin created SPARK-29319:

             Summary: Memory and disk usage not accurate when blocks are evicted and re-loaded
in memory
                 Key: SPARK-29319
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.4, 3.0.0
            Reporter: Marcelo Masiero Vanzin

I found this while running more targeted tests for the underlying behavior of this code, triggered
by SPARK-27468. I ran this code:

import java.util.Arrays
import org.apache.spark.rdd._

def newCachedRDD(level: StorageLevel): RDD[Array[Long]] = {
  val rdd = sc.parallelize(1 to 64, 64).map { i =>
    val a = new Array[Long](1024 * 1024)
    Arrays.fill(a, i)

val r1 = newCachedRDD(level = StorageLevel.MEMORY_AND_DISK)
val r2 = newCachedRDD(level = StorageLevel.MEMORY_ONLY)

With {{./bin/spark-shell --master 'local-cluster[1,1,1024]'}}.

After it runs, you end up with the expected values: r1 has everything cached, only using disk,
because all its memory blocks were evicted by r2; r2 has as many blocks as the memory can

The problem shows up when you start playing with those RDDs again.

Calling {{r1.count()}} will cause all of r2's blocks to be evicted, since r1's blocks are
loaded back in memory. But no block update is sent to the driver about that load, so the driver
does not know that the blocks are now in memory. The UI will show that r1 is using 0 bytes
of memory, and r2 disappears from the storage page (this last one as expected).

Calling {{r2.count()}} after that will cause r1's blocks to be evicted again. This will send
updates to the driver, which will now double-count the disk usage. So if you keep doing this
back and forth, r1's disk usage will keep growing in the UI, when in fact it doesn't change
at all.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message