spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruve Ashar (JIRA)" <>
Subject [jira] [Commented] (SPARK-26827) Support importing python modules having shared objects(.so)
Date Tue, 05 Feb 2019 18:07:00 GMT


Dhruve Ashar commented on SPARK-26827:

Thanks for the response [~irashid] and [~hyukjin.kwon]. Will close this one out.

> Support importing python modules having shared objects(.so)
> -----------------------------------------------------------
>                 Key: SPARK-26827
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: PySpark
>    Affects Versions: 2.3.2, 2.4.0
>            Reporter: Dhruve Ashar
>            Priority: Major
> If a user wants to import dynamic modules, specifically having .so files, this is currently
disallowed by python from a zip file. ([] and
currently spark doesn't support this either. 
> Files which are passed using py-files options are placed on the PYTHONPATH, but are not
extracted. While files which are passed as archives, are extracted but not placed on the PYTHONPATH.
The dynamic modules can be loaded if they are extracted and added to the PYTHONPATH.
> Has anyone encountered this issue before and what is the best way to go about it?
> Some possible solutions:
> 1 - Get around this issue, by passing the archive with py-files and archives option,
this extracts the archive as well as adds it to the path. Gotcha - both have to be named the
same. I have tested this and it works, but its just a workaround.
> 2 - We add a new config like py-archives which takes all the files and extracts them
and also adds them to the PYTHONPATH. Or just examine the contents of the zip file and if
it has dynamic modules then do the same. I am happy to work on the fix.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message