spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luca Canali <Luca.Can...@cern.ch>
Subject Run an OS command or script supplied by the user at the start of each executor
Date Fri, 12 May 2017 18:08:57 GMT
Hi,

I have recently experimented with a few ways to run OS commands from the executors (in a YARN
deployment) for a specific use case where we want to interact with an external system of interest
for our environment. From that experience I thought that having the possibility to spawn a
script at the start of each executors can be quite handy in a few cases and maybe more people
are interested. For example I am think of case when interacting with external systems/APIs,
or for injecting custom configurations via scripts distributed to the executors and/or for
spawning custom monitoring tasks, etc. 
They are probable all niche cases but the feature seems quite easy to implement.
I just wanted to check with the list if something like this has already come up in the past
and/or there are thoughts about it or details that I have overlooked.
My simple proof of concept for implementing a "startup command" on the executors can be found
at:
https://github.com/LucaCanali/spark/commit/e294a1f0d55af115f45fa6d2d7dcf81f751955fa
I can put all this in a Jira in case people here think it makes sense.

Thanks,
Luca


---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Mime
View raw message