I want to step through the work of a spark executor running locally on my machine, from Pycharm.

I am running explicit functionality, in the form of dataset.foreachPartition(f) and I want to see what is going on inside f.

Is there a straightforward way to do it or do I need to resort to remote debugging?


Posted this on SO as well.