spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Plotnikov <pavel.plotni...@team.wrike.com>
Subject Re: Spark diclines mesos offers
Date Wed, 26 Apr 2017 11:53:19 GMT
Michael Gummelt, Thanks!!! I'm forgot about debug logging!

On Mon, Apr 24, 2017 at 9:30 PM Michael Gummelt <mgummelt@mesosphere.io>
wrote:

> Have you run with debug logging?  There are some hints in the debug logs:
> https://github.com/apache/spark/blob/branch-2.1/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L316
>
> On Mon, Apr 24, 2017 at 4:53 AM, Pavel Plotnikov <
> pavel.plotnikov@team.wrike.com> wrote:
>
>> Hi, everyone! I run spark 2.1.0 jobs on the top of Mesos cluster in
>> coarse-grained mode with dynamic resource allocation. And sometimes spark
>> mesos scheduler declines mesos offers despite the fact that not all
>> available resources were used (I have less workers than the possible
>> maximum) and the maximum threshold in the spark configuration is not
>> reached and the queue have lot of pending tasks.
>>
>> May be I have wrong spark or mesos configuration? Does anyone have the
>> same problems?
>>
>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>

Mime
View raw message