spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Braithwaite <a...@cloudflare.com>
Subject Re: Weird Spark Dispatcher Offers?
Date Fri, 02 Oct 2015 19:13:13 GMT
I understand where this is coming from now but I'm still confused as to why
the mesos-master doesn't wait for a framework to ask for resources before
giving it offers.  However, I'm sure I can find that in the mesos docs or
code.

Thanks Tim,
- Alan

On Fri, Oct 2, 2015 at 12:01 PM, Tim Chen <tim@mesosphere.io> wrote:

> Hi Alan,
>
> The dispatcher is getting offers since it's connected to Mesos as a
> framework, as I mentioned earlier all frameworks get offers as long as it's
> connected.
>
> http://mesos.apache.org/documentation/latest/mesos-architecture/
>
> Tim
>
> On Fri, Oct 2, 2015 at 11:58 AM, Alan Braithwaite <alan@cloudflare.com>
> wrote:
>
>> In the Mesos dispatcher case we just decline offers coming in so it's
>>> available for other frameworks.
>>
>>
>> I understand this but do you know why is it getting offers in the first
>> place?
>>
>> Here's the dispatcher startup logs and mesos-master logs right after
>> starting a fresh dispatcher (wiping out the aforementioned directories).
>>
>> https://gist.github.com/anonymous/7e98793d5b3179b6158b
>>
>> And a screenshot of the dispatcher UI to boot:
>>
>> [image: Inline image 1]
>>
>> Thanks,
>> - Alan
>>
>> On Fri, Oct 2, 2015 at 11:55 AM, Tim Chen <tim@mesosphere.io> wrote:
>>
>>> Hi Alan,
>>>
>>> The dispatcher is a Mesos framework and all frameworks in Mesos receives
>>> offers from the master. Mesos is different than most schedulers where
>>> we don't issue containers based on requests, but we offer available
>>> resources to all frameworks and they in turn decide if they want to use
>>> these resources.
>>>
>>> In the Mesos dispatcher case we just decline offers coming in so it's
>>> available for other frameworks.
>>>
>>> Tim
>>>
>>> On Fri, Oct 2, 2015 at 11:51 AM, Alan Braithwaite <alan@cloudflare.com>
>>> wrote:
>>>
>>>> So if there is no jobs to run the dispatcher will decline all offers by
>>>>> default.
>>>>>
>>>>
>>>> So would this be a bug in mesos then?  I'm not sure I understand how
>>>> this offer is appearing in the first place.  It only shows up in the master
>>>> logs when I start the dispatcher.
>>>>
>>>>
>>>>> Also we list all the jobs enqueued and it's specifications in the
>>>>> Spark dispatcher UI, you should see the port in the dispatcher logs itself.
>>>>
>>>>
>>>> Yes, this job is not listed under that UI.  Hence my confusion.
>>>>
>>>> Thanks,
>>>> - Alan
>>>>
>>>> On Fri, Oct 2, 2015 at 11:49 AM, Tim Chen <tim@mesosphere.io> wrote:
>>>>
>>>>> So if there is no jobs to run the dispatcher will decline all offers
>>>>> by default.
>>>>>
>>>>> Also we list all the jobs enqueued and it's specifications in the
>>>>> Spark dispatcher UI, you should see the port in the dispatcher logs itself.
>>>>>
>>>>> Tim
>>>>>
>>>>> On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite <alan@cloudflare.com
>>>>> > wrote:
>>>>>
>>>>>> This happened right after blowing away /var/lib/mesos zk://mesos
and
>>>>>> zk://spark_mesos_dispatcher and before I've submitted anything new
to it so
>>>>>> I _shouldn't_ have anything enqueued.  Unless there's state being
stored
>>>>>> somewhere besides those places that I don't know about.
>>>>>>
>>>>>> I'm not sure what the resource specifications are for this one
>>>>>> because I didn't submit it directly.  If you have a way for me to
grab a
>>>>>> specific offer configuration, I'd be delighted to provide it.  I
just can't
>>>>>> seem to figure out how to get that information after digging through
the
>>>>>> mesos docs :-(
>>>>>>
>>>>>> Also, I can't read the docker logs because:
>>>>>>
>>>>>> Oct 02 11:39:59 sparky docker[556]:
>>>>>> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error
streaming
>>>>>> logs: invalid character '\\x00' looking for beginning of value"
>>>>>>
>>>>>> (that's coming from the spark-dispatcher docker).
>>>>>>
>>>>>> Thanks!
>>>>>> - Alan
>>>>>>
>>>>>> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen <tim@mesosphere.io>
wrote:
>>>>>>
>>>>>>> Do you have jobs enqueued? And if none of the jobs matches any
offer
>>>>>>> it will just decline it.
>>>>>>>
>>>>>>> What's your job resource specifications?
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite <
>>>>>>> alan@cloudflare.com> wrote:
>>>>>>>
>>>>>>>> Hey All,
>>>>>>>>
>>>>>>>> Using spark with mesos and docker.
>>>>>>>>
>>>>>>>> I'm wondering if anybody's seen the behavior of spark dispatcher
>>>>>>>> where it just continually requests resources and immediately
declines the
>>>>>>>> offer.
>>>>>>>>
>>>>>>>> https://gist.github.com/anonymous/41e7c91899b0122b91a7
>>>>>>>>
>>>>>>>> I'm trying to debug some issues with spark and I'm having
trouble
>>>>>>>> figuring out if this is part of the problem or if it's safe
to ignore it.
>>>>>>>>
>>>>>>>> Any help or pointers would be appreciated.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> - Alan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message