spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: Unpersist return type
Date Thu, 22 Oct 2020 13:53:40 GMT
Probably for purposes of chaining, though won't be very useful here. Like
df.unpersist().cache(... some other settings ...)

foreachBatch wants a function that evaluates to Unit, but this qualifies -
doesn't matter what the value of the block is, if it's ignored.
This does seem to compile; are you sure? what error? may not be related to
that, quite.


On Thu, Oct 22, 2020 at 5:40 AM German Schiavon <gschiavonspark@gmail.com>
wrote:

> Hello!
>
> I'd like to ask if there is any reason to return *type *when calling
> *dataframe.unpersist*
>
> def unpersist(blocking: Boolean): this.type = {
>   sparkSession.sharedState.cacheManager.uncacheQuery(
>     sparkSession, logicalPlan, cascade = false, blocking)
>   this
> }
>
>
> Just pointing it out because this example from the docs don't compile
> since unpersist() is not Unit
>
> streamingDF.writeStream.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
>   batchDF.persist()
>   batchDF.write.format(...).save(...)  // location 1
>   batchDF.write.format(...).save(...)  // location 2
>   batchDF.unpersist()}
>
>
> Thanks!
>

Mime
View raw message