spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luca Canali <>
Subject Run an OS command or script supplied by the user at the start of each executor
Date Fri, 12 May 2017 18:08:57 GMT

I have recently experimented with a few ways to run OS commands from the executors (in a YARN
deployment) for a specific use case where we want to interact with an external system of interest
for our environment. From that experience I thought that having the possibility to spawn a
script at the start of each executors can be quite handy in a few cases and maybe more people
are interested. For example I am think of case when interacting with external systems/APIs,
or for injecting custom configurations via scripts distributed to the executors and/or for
spawning custom monitoring tasks, etc. 
They are probable all niche cases but the feature seems quite easy to implement.
I just wanted to check with the list if something like this has already come up in the past
and/or there are thoughts about it or details that I have overlooked.
My simple proof of concept for implementing a "startup command" on the executors can be found
I can put all this in a Jira in case people here think it makes sense.


To unsubscribe e-mail:

View raw message