spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dzcxzl (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-29657) Iterator spill supporting radix sort with null prefix
Date Wed, 30 Oct 2019 11:13:00 GMT
dzcxzl created SPARK-29657:
------------------------------

             Summary: Iterator spill supporting radix sort with null prefix
                 Key: SPARK-29657
                 URL: https://issues.apache.org/jira/browse/SPARK-29657
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: dzcxzl


In the case of radix sort, when the insertRecord part of the keyPrefix is null, the iterator
type returned by getSortedIterator is ChainedIterator.
Currently ChainedIterator does not support spill, causing UnsafeExternalSorter to take up
a lot of execution memory, allocatePage fails, throw SparkOutOfMemoryError Unable to acquire
xxx bytes of memory, got 0

The following is a log of an error we encountered in the production environment.

[Executor task launch worker for task 66055] INFO TaskMemoryManager: Memory used in task 66055
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@39dd866e:
64.0 KB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@74d17927:
4.6 GB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@31478f9c:
61.0 MB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: 0 bytes of memory were
used by task 66055 but are not associated with specific consumers
[Executor task launch worker for task 66055] INFO TaskMemoryManager: 4962998749 bytes of memory
are used for execution and 2218326 bytes of memory are used for storage
[Executor task launch worker for task 66055] ERROR Executor: Exception in task 42.3 in stage
29.0 (TID 66055)
SparkOutOfMemoryError: Unable to acquire 3436 bytes of memory, got 0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message