spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Erlandson <>
Subject Re: Remove non-Tungsten mode in Spark 3?
Date Thu, 10 Jan 2019 00:06:03 GMT
Removing the user facing config seems like a good idea from the standpoint
of reducing cognitive load, and documentation

On Fri, Jan 4, 2019 at 7:03 AM Sean Owen <> wrote:

> OK, maybe leave in tungsten for 3.0.
> I did a quick check, and removing StaticMemoryManager saves a few hundred
> lines. It's used in MemoryStore tests internally though, and not a trivial
> change to remove it. It's also used directly in HashedRelation. It could
> still be worth removing it as a user-facing option to reduce confusion
> about memory tuning, but it wouldn't take out much code. What do you all
> think?
> On Thu, Jan 3, 2019 at 9:41 PM Reynold Xin <> wrote:
>> The issue with the offheap mode is it is a pretty big behavior change and
>> does require additional setup (also for users that run with UDFs that
>> allocate a lot of heap memory, it might not be as good).
>> I can see us removing the legacy mode since it's been legacy for a long
>> time and perhaps very few users need it. How much code does it remove
>> though?
>> On Thu, Jan 03, 2019 at 2:55 PM, Sean Owen <> wrote:
>>> Just wondering if there is a good reason to keep around the pre-tungsten
>>> on-heap memory mode for Spark 3, and make spark.memory.offHeap.enabled
>>> always true? It would simplify the code somewhat, but I don't feel I'm so
>>> aware of the tradeoffs.
>>> I know we didn't deprecate it, but it's been off by default for a long
>>> time. It could be deprecated, too.
>>> Same question for spark.memory.useLegacyMode and all its various
>>> associated settings? Seems like these should go away at some point, and
>>> Spark 3 is a good point. Same issue about deprecation though.
>>> --------------------------------------------------------------------- To
>>> unsubscribe e-mail:

View raw message