spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajani, Arpan" <>
Subject RE: releasing memory without stopping the spark context ?
Date Wed, 31 Aug 2016 16:18:34 GMT
Removing Data

Spark automatically monitors cache usage on each node and drops out old data partitions in
a least-recently-used (LRU) fashion.
If you would like to manually remove an RDD instead of waiting for it to fall out of the cache,
use the RDD.unpersist() method. (Copied from Documentation<>

From: Cesar []
Sent: 31 August 2016 16:57
To: user
Subject: releasing memory without stopping the spark context ?

Is there a way to release all persisted RDD's/DataFrame's in Spark without stopping the SparkContext

Thanks a lot
Cesar Flores
This e-mail and any attachments are confidential, intended only for the addressee and may
be privileged. If you have received this e-mail in error, please notify the sender immediately
and delete it. Any content that does not relate to the business of Worldpay is personal to
the sender and not authorised or endorsed by Worldpay. Worldpay does not accept responsibility
for viruses or any loss or damage arising from transmission or access.

Worldpay (UK) Limited (Company No: 07316500/ Financial Conduct Authority No: 530923), Worldpay
Limited (Company No:03424752 / Financial Conduct Authority No: 504504), Worldpay AP Limited
(Company No: 05593466 / Financial Conduct Authority No: 502597). Registered Office: The Walbrook
Building, 25 Walbrook, London EC4N 8AF and authorised by the Financial Conduct Authority under
the Payment Service Regulations 2009 for the provision of payment services. Worldpay (UK)
Limited is authorised and regulated by the Financial Conduct Authority for consumer credit
activities. Worldpay B.V. (WPBV) has its registered office in Amsterdam, the Netherlands (Handelsregister
KvK no. 60494344). WPBV holds a licence from and is included in the register kept by De Nederlandsche
Bank, which registration can be consulted through Worldpay, the logo and any associated
brand names are trade marks of the Worldpay group.
View raw message