spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Utkarsh Sengar <utkarsh2...@gmail.com>
Subject Re: Using accumulator to push custom logs to driver
Date Mon, 01 Feb 2016 23:48:34 GMT
Not alot of string messages, I need it mostly for debugging purposed which
I will use on an ahdoc basis - manually add debug statements which returns
info about the dataset etc.
I would assume the strings will vary from 100-200lines max, that would be
about 50-100KB if they are really long lines.

-Utkarsh

On Mon, Feb 1, 2016 at 3:40 PM, Holden Karau <holden@pigscanfly.ca> wrote:

> I wouldn't use accumulators for things which could get large, they can
> become kind of a bottle neck. Do you have a lot of string messages you want
> to bring back or only a few?
>
> On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <utkarsh2012@gmail.com>
> wrote:
>
>> I am trying to debug code executed in executors by logging. Even when I
>> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in
>> the corresponding slaves.
>> Its anyway inefficient to keep checking multiple slaves for logs.
>>
>> One way to deal with this is to push logs to a central location.
>>
>> Another way (for debugging purposes) is to use accumulators . Is it
>> advisable to use accumulators to push string from executors to driver?
>> It will simplify things when I am debugging datasets, bugs which is hard
>> to reproduce locally etc.
>>
>> Suggestions/comments?
>>
>> -Utkarsh
>>
>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
>



-- 
Thanks,
-Utkarsh

Mime
View raw message