spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeffrey Jedele <>
Subject Re: Spark per app logging
Date Sat, 21 Mar 2015 10:13:58 GMT
I'm not completely sure about this either, but this is what we are doing
Configure your logging to write to STDOUT, not to a file explicitely. Spark
will capture stdour and stderr and separate the messages into a app/driver
folder structure in the configured worker directory.

We then use logstash to collect the logs and index them to a elasticsearch
cluster (Spark seems to produce a lot of logging data). With some simple
regex processing, you also get the application id as searchable field.


2015-03-20 22:37 GMT+01:00 Ted Yu <>:

> Are these jobs the same jobs, just run by different users or, different
> jobs ?
> If the latter, can each application use its own ?
> Cheers
> On Fri, Mar 20, 2015 at 1:43 PM, Udit Mehta <> wrote:
>> Hi,
>> We have spark setup such that there are various users running multiple
>> jobs at the same time. Currently all the logs go to 1 file specified in the
>> Is it possible to configure log4j in spark for per app/user logging
>> instead of sending all logs to 1 file mentioned in the
>> Thanks
>> Udit

View raw message