spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Using accumulator to push custom logs to driver
Date Mon, 01 Feb 2016 23:40:32 GMT
I wouldn't use accumulators for things which could get large, they can
become kind of a bottle neck. Do you have a lot of string messages you want
to bring back or only a few?

On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <utkarsh2012@gmail.com>
wrote:

> I am trying to debug code executed in executors by logging. Even when I
> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in
> the corresponding slaves.
> Its anyway inefficient to keep checking multiple slaves for logs.
>
> One way to deal with this is to push logs to a central location.
>
> Another way (for debugging purposes) is to use accumulators . Is it
> advisable to use accumulators to push string from executors to driver?
> It will simplify things when I am debugging datasets, bugs which is hard
> to reproduce locally etc.
>
> Suggestions/comments?
>
> -Utkarsh
>



-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

Mime
View raw message