spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Afshin, Bardia" <>
Subject Spark-submit Py-files with EMR add step?
Date Wed, 07 Mar 2018 18:21:34 GMT
I’m writing this email to reach out to the community to demisty the py-files parameter when
working with spark-submit and python projects.

Currently I have a project, say


  *   Modules/

When I zip up the src directory and submit it to spark via emr add step , the namespacing
is lost. example:
From Modules.module1 import SomeClass

My code returns and error that it cannot find this class, now this works if I goto the instance
download my project, and submit it to spark from within the EMR instance via spark-submit
, but not when adding it as a step in emr from external call.


This message is confidential, intended only for the named recipient(s) and may contain information
that is privileged or exempt from disclosure under applicable law. If you are not the intended
recipient(s), you are notified that the dissemination, distribution, or copying of this message
is strictly prohibited. If you receive this message in error or are not the named recipient(s),
please notify the sender by return email and delete this message. Thank you.
View raw message