spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-15357) Cooperative spilling should check consumer memory mode
Date Mon, 16 May 2016 23:55:12 GMT

     [ https://issues.apache.org/jira/browse/SPARK-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or updated SPARK-15357:
------------------------------
    Description: 
In TaskMemoryManager.java:
{code}
for (MemoryConsumer c: consumers) {
  if (c != consumer && c.getUsed() > 0) {
    try {
      long released = c.spill(required - got, consumer);
        if (released > 0 && mode == tungstenMemoryMode) {
          got += memoryManager.acquireExecutionMemory(required - got, taskAttemptId, mode);
          if (got >= required) {
            break;
          }
        }
      } catch(...) { ... }
    }
  }
}
{code}
Currently, when non-tungsten consumers acquire execution memory, they may force other tungsten
consumers to spill and then NOT use the freed memory. A better way to do this is to incorporate
the memory mode in the consumer itself and spill only those with matching memory modes.

  was:
In TaskMemoryManager.java:
{code}
for (MemoryConsumer c: consumers) {
          if (c != consumer && c.getUsed() > 0) {
            try {
              long released = c.spill(required - got, consumer);
              if (released > 0 && mode == tungstenMemoryMode) {
                logger.debug("Task {} released {} from {} for {}", taskAttemptId,
                  Utils.bytesToString(released), c, consumer);
                got += memoryManager.acquireExecutionMemory(required - got, taskAttemptId,
mode);
                if (got >= required) {
                  break;
                }
              }
            } catch (IOException e) { ... }
          }
        }
{code}
Currently, when non-tungsten consumers acquire execution memory, they may force other tungsten
consumers to spill and then NOT use the freed memory. A better way to do this is to incorporate
the memory mode in the consumer itself and spill only those with matching memory modes.


> Cooperative spilling should check consumer memory mode
> ------------------------------------------------------
>
>                 Key: SPARK-15357
>                 URL: https://issues.apache.org/jira/browse/SPARK-15357
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0
>            Reporter: Andrew Or
>
> In TaskMemoryManager.java:
> {code}
> for (MemoryConsumer c: consumers) {
>   if (c != consumer && c.getUsed() > 0) {
>     try {
>       long released = c.spill(required - got, consumer);
>         if (released > 0 && mode == tungstenMemoryMode) {
>           got += memoryManager.acquireExecutionMemory(required - got, taskAttemptId,
mode);
>           if (got >= required) {
>             break;
>           }
>         }
>       } catch(...) { ... }
>     }
>   }
> }
> {code}
> Currently, when non-tungsten consumers acquire execution memory, they may force other
tungsten consumers to spill and then NOT use the freed memory. A better way to do this is
to incorporate the memory mode in the consumer itself and spill only those with matching memory
modes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message