spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <>
Subject Re: Using accumulator to push custom logs to driver
Date Mon, 01 Feb 2016 23:40:32 GMT
I wouldn't use accumulators for things which could get large, they can
become kind of a bottle neck. Do you have a lot of string messages you want
to bring back or only a few?

On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <>

> I am trying to debug code executed in executors by logging. Even when I
> add log4j's inside .map() I don't see it in mesos task logs in
> the corresponding slaves.
> Its anyway inefficient to keep checking multiple slaves for logs.
> One way to deal with this is to push logs to a central location.
> Another way (for debugging purposes) is to use accumulators . Is it
> advisable to use accumulators to push string from executors to driver?
> It will simplify things when I am debugging datasets, bugs which is hard
> to reproduce locally etc.
> Suggestions/comments?
> -Utkarsh

Cell : 425-233-8271

View raw message