spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mkal <>
Subject Question about RDD pipe
Date Thu, 17 Jan 2019 22:09:25 GMT
Hi, im trying to run an external script on spark using rdd.pipe() and
although it runs successfully on standalone, it throws an error on cluster.
The error comes from the executors and it's : "Cannot run program
"path/to/program": error=2, No such file or directory".

Does the external script need to be available on all nodes in the cluster
when using rdd.pipe()?

What if i don't have permission to install anything on the nodes of the
cluster? Is there any other way to make the script available to the worker

(The external script is loaded in HDFS and is passed to the driver class
through args)

Sent from:

To unsubscribe e-mail:

View raw message