spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Getting the size of a broadcast variable
Date Tue, 02 Feb 2016 08:35:42 GMT
There is chance that the log message may change in future releases.

Log snooping would be broken.

FYI

On Mon, Feb 1, 2016 at 9:55 PM, Takeshi Yamamuro <linguin.m.s@gmail.com>
wrote:

> Hi,
>
> Currently, there is no way to check the size except for snooping INFO-logs
> in a driver;
>
> 16/02/02 14:51:53 INFO BlockManagerInfo: Added rdd_2_12 in memory on
> localhost:58536 (size: 40.0 B, free: 510.7 MB)
>
>
>
> On Tue, Feb 2, 2016 at 8:20 AM, apu mishra . rr <apumishra.rr@gmail.com>
> wrote:
>
>> How can I determine the size (in bytes) of a broadcast variable? Do I
>> need to use the .dump method and then look at the size of the result, or is
>> there an easier way?
>>
>> Using PySpark with Spark 1.6.
>>
>> Thanks!
>>
>> Apu
>>
>
>
>
> --
> ---
> Takeshi Yamamuro
>

Mime
View raw message