spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Szuromi <tamas.szur...@odigeo.com>
Subject Re: Spark, Mesos problems with remote connections
Date Mon, 02 Nov 2015 14:31:38 GMT
Hello Sebastian,

Did you set the MESOS_NATIVE_JAVA_LIBRARY variable before you started
pyspark?

cheers,

Tamas




On 2 November 2015 at 15:24, Sebastian Kuepers <
sebastian.kuepers@publicispixelpark.de> wrote:

> Hey,
>
>
> I have a Mesos cluster with a single Master. If I run the following
> directly on the master machine:
>
>
> pyspark --master mesos://host:5050
>
>
> everything works just fine. If I try to connect from to the master
> starting a driver from my laptop everything stops after the following
> log output from the Spark driver:
>
>
> I1102 15:10:57.848831 64856064 sched.cpp:164] Version: 0.24.0
> I1102 15:10:57.852708 19017728 sched.cpp:262] New master detected at
> master@217.66.55.19:5050
> I1102 15:10:57.852934 19017728 sched.cpp:272] No credentials provided.
> Attempting to register without authentication
>
>
> The Mesos master logs show the following:
>
>
> I1102 15:09:50.676004 21563 master.cpp:2250] Subscribing framework Talos
> with checkpointing disabled and capabilities [  ]
> I1102 15:09:50.676686 21566 hierarchical.hpp:515] Added framework
> b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004
> I1102 15:09:51.671437 21562 http.cpp:336] HTTP GET for /master/state.json
> from 217.66.51.150:51588 with User-Agent='Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36'
> I1102 15:09:52.357230 21567 master.cpp:2179] Received SUBSCRIBE call for
> framework 'Talos' at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> I1102 15:09:52.357416 21567 master.cpp:2250] Subscribing framework Talos
> with checkpointing disabled and capabilities [  ]
> I1102 15:09:52.357520 21567 master.cpp:2260] Framework
> b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004 (Talos) at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> already subscribed, resending acknowledgement
> I1102 15:09:53.938227 21563 master.cpp:2179] Received SUBSCRIBE call for
> framework 'Talos' at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> I1102 15:09:53.938426 21563 master.cpp:2250] Subscribing framework Talos
> with checkpointing disabled and capabilities [  ]
> I1102 15:09:53.938534 21563 master.cpp:2260] Framework
> b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004 (Talos) at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> already subscribed, resending acknowledgement
> I1102 15:10:00.207372 21567 master.cpp:2179] Received SUBSCRIBE call for
> framework 'Talos' at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
>
>
> The framework also shows up as active in the UI and blocks all resources
> of the cluster, while the SUBSCRIBE calls keep coming in.
>
> Mesos authentication is completely disabled.
>
>
> What could be possible causes for this problem?
>
>
> Thanks,
>
> Sebastian
>
>
>
>
> ------------------------------------------------------------------------
> Disclaimer The information in this email and any attachments may contain
> proprietary and confidential information that is intended for the
> addressee(s) only. If you are not the intended recipient, you are hereby
> notified that any disclosure, copying, distribution, retention or use of
> the contents of this information is prohibited. When addressed to our
> clients or vendors, any information contained in this e-mail or any
> attachments is subject to the terms and conditions in any governing
> contract. If you have received this e-mail in error, please immediately
> contact the sender and delete the e-mail.
>

Mime
View raw message