spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Kronenfeld <nkronenf...@uncharted.software>
Subject Re: isCached
Date Fri, 01 Sep 2017 16:08:38 GMT
Thanks for the info

On Fri, Sep 1, 2017 at 12:06 PM, Nick Pentreath <nick.pentreath@gmail.com>
wrote:

> No unfortunately not - as i recall storageLevel accesses some private
> methods to get the result.
>
> On Fri, 1 Sep 2017 at 17:55, Nathan Kronenfeld <nkronenfeld@uncharted.software>
> wrote:
>
>> Ah, in 2.1.0.
>>
>> I'm in 2.0.1 at the moment... is there any way that works that far back?
>>
>> On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <nick.pentreath@gmail.com
>> > wrote:
>>
>>> Dataset does have storageLevel. So you can use isCached = (storageLevel
>>> != StorageLevel.NONE) as a test.
>>>
>>> Arguably isCached could be added to dataset too, shouldn't be a
>>> controversial change.
>>>
>>> On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld <nkronenfeld@uncharted.software>
>>> wrote:
>>>
>>>> I'm currently porting some of our code from RDDs to Datasets.
>>>>
>>>> With RDDs it's pretty easy to figure out if they are cached or not.
>>>>
>>>> I notice that the catalog has a function for determining this on
>>>> Datasets too, but it's private[sql].  Is there any reason for it not to be
>>>> public?  Is there any way at the moment to determine if a dataset is cached
>>>> or not?
>>>>
>>>> Thanks in advance
>>>>                -Nathan Kronenfeld
>>>>
>>>
>>

Mime
View raw message