spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Braithwaite <a...@cloudflare.com>
Subject Re: Weird Spark Dispatcher Offers?
Date Fri, 02 Oct 2015 18:58:54 GMT
>
> In the Mesos dispatcher case we just decline offers coming in so it's
> available for other frameworks.


I understand this but do you know why is it getting offers in the first
place?

Here's the dispatcher startup logs and mesos-master logs right after
starting a fresh dispatcher (wiping out the aforementioned directories).

https://gist.github.com/anonymous/7e98793d5b3179b6158b

And a screenshot of the dispatcher UI to boot:

[image: Inline image 1]

Thanks,
- Alan

On Fri, Oct 2, 2015 at 11:55 AM, Tim Chen <tim@mesosphere.io> wrote:

> Hi Alan,
>
> The dispatcher is a Mesos framework and all frameworks in Mesos receives
> offers from the master. Mesos is different than most schedulers where
> we don't issue containers based on requests, but we offer available
> resources to all frameworks and they in turn decide if they want to use
> these resources.
>
> In the Mesos dispatcher case we just decline offers coming in so it's
> available for other frameworks.
>
> Tim
>
> On Fri, Oct 2, 2015 at 11:51 AM, Alan Braithwaite <alan@cloudflare.com>
> wrote:
>
>> So if there is no jobs to run the dispatcher will decline all offers by
>>> default.
>>>
>>
>> So would this be a bug in mesos then?  I'm not sure I understand how this
>> offer is appearing in the first place.  It only shows up in the master logs
>> when I start the dispatcher.
>>
>>
>>> Also we list all the jobs enqueued and it's specifications in the Spark
>>> dispatcher UI, you should see the port in the dispatcher logs itself.
>>
>>
>> Yes, this job is not listed under that UI.  Hence my confusion.
>>
>> Thanks,
>> - Alan
>>
>> On Fri, Oct 2, 2015 at 11:49 AM, Tim Chen <tim@mesosphere.io> wrote:
>>
>>> So if there is no jobs to run the dispatcher will decline all offers by
>>> default.
>>>
>>> Also we list all the jobs enqueued and it's specifications in the Spark
>>> dispatcher UI, you should see the port in the dispatcher logs itself.
>>>
>>> Tim
>>>
>>> On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite <alan@cloudflare.com>
>>> wrote:
>>>
>>>> This happened right after blowing away /var/lib/mesos zk://mesos and
>>>> zk://spark_mesos_dispatcher and before I've submitted anything new to it
so
>>>> I _shouldn't_ have anything enqueued.  Unless there's state being stored
>>>> somewhere besides those places that I don't know about.
>>>>
>>>> I'm not sure what the resource specifications are for this one because
>>>> I didn't submit it directly.  If you have a way for me to grab a specific
>>>> offer configuration, I'd be delighted to provide it.  I just can't seem to
>>>> figure out how to get that information after digging through the mesos docs
>>>> :-(
>>>>
>>>> Also, I can't read the docker logs because:
>>>>
>>>> Oct 02 11:39:59 sparky docker[556]:
>>>> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming
>>>> logs: invalid character '\\x00' looking for beginning of value"
>>>>
>>>> (that's coming from the spark-dispatcher docker).
>>>>
>>>> Thanks!
>>>> - Alan
>>>>
>>>> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen <tim@mesosphere.io> wrote:
>>>>
>>>>> Do you have jobs enqueued? And if none of the jobs matches any offer
>>>>> it will just decline it.
>>>>>
>>>>> What's your job resource specifications?
>>>>>
>>>>> Tim
>>>>>
>>>>> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite <alan@cloudflare.com
>>>>> > wrote:
>>>>>
>>>>>> Hey All,
>>>>>>
>>>>>> Using spark with mesos and docker.
>>>>>>
>>>>>> I'm wondering if anybody's seen the behavior of spark dispatcher
>>>>>> where it just continually requests resources and immediately declines
the
>>>>>> offer.
>>>>>>
>>>>>> https://gist.github.com/anonymous/41e7c91899b0122b91a7
>>>>>>
>>>>>> I'm trying to debug some issues with spark and I'm having trouble
>>>>>> figuring out if this is part of the problem or if it's safe to ignore
it.
>>>>>>
>>>>>> Any help or pointers would be appreciated.
>>>>>>
>>>>>> Thanks!
>>>>>> - Alan
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message