spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phillip Henry <londonjava...@gmail.com>
Subject Re: Adding Custom finalize method to RDDs.
Date Thu, 13 Jun 2019 09:27:35 GMT
If you control the codebase, you control when an RDD goes out of scope. Or
am I missing something?

(Note that finalize will not necessarily executed when an object goes out
of scope but when the GC runs at some indeterminate point in the future.
Please avoid using finalize for the kind of task you're trying to do. It's
not what it was designed for. Better to pay more attention to
house-keeping in your own code.)



On Wed, Jun 12, 2019 at 9:11 PM Nasrulla Khan Haris <
Nasrulla.Khan@microsoft.com> wrote:

> We cannot have control over RDD going out of scope from memory as it was
> handled by JVM. Thus I am not sure try and finalize will help.
>
>  Thus I wanted to use some mechanism to cleanup of some temporary data
> which is created by RDD immediately as soon as it goes out of scope.
>
>
>
> Any ideas ?
>
>
>
> Thanks,
>
> Nasrulla
>
>
>
> *From:* Phillip Henry <londonjavaman@gmail.com>
> *Sent:* Tuesday, June 11, 2019 11:28 PM
> *To:* Nasrulla Khan Haris <Nasrulla.Khan@microsoft.com.invalid>
> *Cc:* Vinoo Ganesh <vganesh@palantir.com>; dev@spark.apache.org
> *Subject:* Re: Adding Custom finalize method to RDDs.
>
>
>
> That's not the kind of thing a finalize method was ever supposed to do.
>
>
>
> Use a try/finally block instead.
>
>
>
> Phillip
>
>
>
>
>
> On Wed, 12 Jun 2019, 00:01 Nasrulla Khan Haris, <
> Nasrulla.Khan@microsoft.com.invalid> wrote:
>
> I want to delete some files which I created In my datasource api,  as soon
> as the RDD is cleaned up.
>
>
>
> Thanks,
>
> Nasrulla
>
>
>
> *From:* Vinoo Ganesh <vganesh@palantir.com>
> *Sent:* Monday, June 10, 2019 1:32 PM
> *To:* Nasrulla Khan Haris <Nasrulla.Khan@microsoft.com.INVALID>;
> dev@spark.apache.org
> *Subject:* Re: Adding Custom finalize method to RDDs.
>
>
>
> Generally overriding the finalize() method is an antipattern (it was in
> fact deprecated in java 11
> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize())
> . What’s the use case here?
>
>
>
> *From: *Nasrulla Khan Haris <Nasrulla.Khan@microsoft.com.INVALID>
> *Date: *Monday, June 10, 2019 at 15:44
> *To: *"dev@spark.apache.org" <dev@spark.apache.org>
> *Subject: *RE: Adding Custom finalize method to RDDs.
>
>
>
> Hello Everyone,
>
> Is there a way  to do it from user-code ?
>
>
>
> Thanks,
>
> Nasrulla
>
>
>
> *From:* Nasrulla Khan Haris <Nasrulla.Khan@microsoft.com.INVALID>
> *Sent:* Sunday, June 9, 2019 5:30 PM
> *To:* dev@spark.apache.org
> *Subject:* Adding Custom finalize method to RDDs.
>
>
>
> Hi All,
>
>
>
> Is there a way to add custom finalize method to RDD objects to add custom
> logic when RDDs are destructed by JVM ?
>
>
>
> Thanks,
>
> Nasrulla
>
>
>
>

Mime
View raw message