spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Using accumulator to push custom logs to driver
Date Tue, 02 Feb 2016 00:49:44 GMT
Ah if its manual ad-hoc logging of the 100 to 200 lines then thats probably
OK.

On Mon, Feb 1, 2016 at 3:48 PM, Utkarsh Sengar <utkarsh2012@gmail.com>
wrote:

> Not alot of string messages, I need it mostly for debugging purposed which
> I will use on an ahdoc basis - manually add debug statements which returns
> info about the dataset etc.
> I would assume the strings will vary from 100-200lines max, that would be
> about 50-100KB if they are really long lines.
>
> -Utkarsh
>
> On Mon, Feb 1, 2016 at 3:40 PM, Holden Karau <holden@pigscanfly.ca> wrote:
>
>> I wouldn't use accumulators for things which could get large, they can
>> become kind of a bottle neck. Do you have a lot of string messages you want
>> to bring back or only a few?
>>
>> On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <utkarsh2012@gmail.com>
>> wrote:
>>
>>> I am trying to debug code executed in executors by logging. Even when I
>>> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in
>>> the corresponding slaves.
>>> Its anyway inefficient to keep checking multiple slaves for logs.
>>>
>>> One way to deal with this is to push logs to a central location.
>>>
>>> Another way (for debugging purposes) is to use accumulators . Is it
>>> advisable to use accumulators to push string from executors to driver?
>>> It will simplify things when I am debugging datasets, bugs which is hard
>>> to reproduce locally etc.
>>>
>>> Suggestions/comments?
>>>
>>> -Utkarsh
>>>
>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>>
>
>
>
> --
> Thanks,
> -Utkarsh
>



-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

Mime
View raw message