spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Utkarsh Sengar <>
Subject Re: Using accumulator to push custom logs to driver
Date Mon, 01 Feb 2016 23:48:34 GMT
Not alot of string messages, I need it mostly for debugging purposed which
I will use on an ahdoc basis - manually add debug statements which returns
info about the dataset etc.
I would assume the strings will vary from 100-200lines max, that would be
about 50-100KB if they are really long lines.


On Mon, Feb 1, 2016 at 3:40 PM, Holden Karau <> wrote:

> I wouldn't use accumulators for things which could get large, they can
> become kind of a bottle neck. Do you have a lot of string messages you want
> to bring back or only a few?
> On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <>
> wrote:
>> I am trying to debug code executed in executors by logging. Even when I
>> add log4j's inside .map() I don't see it in mesos task logs in
>> the corresponding slaves.
>> Its anyway inefficient to keep checking multiple slaves for logs.
>> One way to deal with this is to push logs to a central location.
>> Another way (for debugging purposes) is to use accumulators . Is it
>> advisable to use accumulators to push string from executors to driver?
>> It will simplify things when I am debugging datasets, bugs which is hard
>> to reproduce locally etc.
>> Suggestions/comments?
>> -Utkarsh
> --
> Cell : 425-233-8271
> Twitter:


View raw message