spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17336) Repeated calls sbin/spark-config.sh file Causes ${PYTHONPATH} Value duplicate
Date Thu, 01 Sep 2016 08:22:22 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454745#comment-15454745
] 

Sean Owen commented on SPARK-17336:
-----------------------------------

[~axu4apache] I assume you modified the scripts to print PYTHONPATH in order to show this
behavior.
Yeah it looks like the file that sets it is repeatedly sourced so this path is appended many
times.
I don't know of a cleaner way to do this other than to see if the variable already contains
the path and only append if it doesn't. Do you want to try that in a PR?

> Repeated calls sbin/spark-config.sh file Causes ${PYTHONPATH} Value duplicate
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-17336
>                 URL: https://issues.apache.org/jira/browse/SPARK-17336
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.0.0
>            Reporter: anxu
>
> On Spark start up by command: sbin/start-all.sh, the sbin/spark-config.sh Repeated calls.
In sbin/spark-config.sh code.
> {code:title=sbin/spark-config.sh|borderStyle=solid}
> # Add the PySpark classes to the PYTHONPATH:
> export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
> export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
> {code}
> {color:red}PYTHONPATH{color} has duplicate Value.
> example:
> {code:borderStyle=solid}
> axu4iMac:spark-2.0.0-hadoop2.4 axu$ sbin/start-all.sh  | grep PYTHONPATH
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message