spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean R. Owen (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-29077) submitting a SparkSession job fails on a spark://localhost:7077 url on Mac
Date Sat, 26 Oct 2019 22:57:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-29077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960457#comment-16960457
] 

Sean R. Owen commented on SPARK-29077:
--------------------------------------

You'll need to open a pull request.

> submitting  a SparkSession job fails on a spark://localhost:7077 url on Mac
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-29077
>                 URL: https://issues.apache.org/jira/browse/SPARK-29077
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.4.3
>         Environment: Darwin <hostname.local> 18.7.0 Darwin Kernel Version 18.7.0:
Thu Jun 20 
>  PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64
>            Reporter: alex
>            Priority: Major
>
> When creating a spark context running on a local host getting connection refused error,
however using the actual host name for example: "spark://myhostname.local:7077" works
> {code:java}
> performance-meter {
>     spark {
>         appname = "test-harness"
>         master = "spark://localhost:7077"
>     }
> } {code}
> {code:java}
> val configRoot = "performance-meter"
> val sparkSession = SparkSession.builder
>   .appName(conf.getString(s"${configRoot}.spark.appname"))
>   .master(conf.getString(s"${configRoot}.spark.master")) {code}
>  
> This appears to be due to some Macs having multiple network interfaces, at least is the
case on my Mac. Recommended fix that seems to work locally:
> in file: /usr/local/Cellar/apache-spark/2.4.3/libexec/sbin/start-master.sh & /usr/local/Cellar/apache-spark/2.4.3/libexec/sbin/start-slaves.sh add
a case section for "Darwin"
> {code:bash}
> if [ "$SPARK_MASTER_HOST" = "" ]; then
>   case `uname` in
>       (SunOS)
>     SPARK_MASTER_HOST="`/usr/sbin/check-hostname | awk '{print $NF}'`"
>     ;;
>      (Darwin)
>     SPARK_MASTER_HOST="localhost" # 13-Sep-2019 alexshagiev add Mac (Darwin) case to
ensure spark binds on local host interface instead of external interface.
>     ;;
>       (*)
>     SPARK_MASTER_HOST="`hostname -f`"
>     ;;
>   esac
> fi
> {code}
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message