spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From flyingmeatball <>
Subject Spark 2.4.0 worker can't find work/app/folderNo directory for logs
Date Thu, 29 Nov 2018 19:03:21 GMT
I'm running a 10 node standalone cluster and I'm having issues with a stage
completing - it keeps hanging somewhere between 196 and 199/200 blocks
completed, but never errors and doesn't move forward.


If I look at the task(s) still running, the stdout and stderr always give
the same message:
Error: invalid log directory


This always happens on the same node. If I SSH into that node app folder, I
see that there is a /1/, but not a /0/.

Why is it looking for the wrong folder? This is stage 16/19, so it isn't
like it bombs from the get-go - that executor has done many previous tasks.
I can't figure out how to troubleshoot any further - the spark job never
bombs, that one task just keeps running...



Sent from:

To unsubscribe e-mail:

View raw message