spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirav Patel <>
Subject Stage or Tasks level logs missing
Date Wed, 13 Feb 2019 18:54:26 GMT
Currently there seems to be 3 places to check task level logs:
1) Using spark UI
2) `yarn application log`
3) log aggregation  on hdfs (if enabled)

All above only give you log at executor(container) level. However one
executor can have multiple threads and each might be running part of
different stages(stg1, stg2)) and within that different tasks(tid1,
tid2...) . It's hard to track particular task activities  in Executor logs.

It'd be nice if:
1) mark all log entries with stageId followed by taskId
2) have a separate log file for each task (taskId)
3) have a separated log file for of stage level logs

If I missed something let me know


View raw message