spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Ippolitov <>
Subject [YarnShuffleService] Consistent OOMs when enabling Spark transport encryption
Date Mon, 28 Oct 2019 10:09:17 GMT

I have been experimenting with Spark 2.4.4 transport encryption and have
encountered an issue with a couple of our jobs: they consistently make the
YarnShuffleService die with OOM errors. It looks like the memory is full of
/$Entry/ objects each containing
/$EncryptedMessage/ objects. 

Someone else has created a Jira ticket (SPARK-28743) for this a couple of
months ago but there was no answer. I have shared more details in the ticket

- Has anyone experienced anything like this as well?
- Is there anyone with knowledge of the YarnShuffleService, and/or Netty
and/or Spark encryption that would be able to lend a hand in debugging this


Sent from:

To unsubscribe e-mail:

View raw message