spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fairiz Azizi <code...@gmail.com>
Subject Re: Spark on Mesos 0.20
Date Wed, 08 Oct 2014 00:36:42 GMT
Sure, could you point me to the example?

The only thing I could find was
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/LogQuery.scala

So do you mean running it like:
   MASTER="mesos://xxxxxxx*:5050*" ./run-example LogQuery

I tried that and I can see the job run and the tasks complete on the slave
nodes, but the client process seems to hang forever, it's probably a
different problem. BTW, only a dozen or so tasks kick off.

I actually haven't done much with Scala and Spark (it's been all python).

Fi



Fairiz "Fi" Azizi

On Tue, Oct 7, 2014 at 6:29 AM, RJ Nowling <rnowling@gmail.com> wrote:

> I was able to reproduce it on a small 4 node cluster (1 mesos master and 3
> mesos slaves) with relatively low-end specs.  As I said, I just ran the log
> query examples with the fine-grained mesos mode.
>
> Spark 1.1.0 and mesos 0.20.1.
>
> Fairiz, could you try running the logquery example included with Spark and
> see what you get?
>
> Thanks!
>
> On Mon, Oct 6, 2014 at 8:07 PM, Fairiz Azizi <coderfi@gmail.com> wrote:
>
>> That's what great about Spark, the community is so active! :)
>>
>> I compiled Mesos 0.20.1 from the source tarball.
>>
>> Using the Mapr3 Spark 1.1.0 distribution from the Spark downloads page
>>  (spark-1.1.0-bin-mapr3.tgz).
>>
>> I see no problems for the workloads we are trying.
>>
>> However, the cluster is small (less than 100 cores across 3 nodes).
>>
>> The workloads reads in just a few gigabytes from HDFS, via an ipython
>> notebook spark shell.
>>
>> thanks,
>> Fi
>>
>>
>>
>> Fairiz "Fi" Azizi
>>
>> On Mon, Oct 6, 2014 at 9:20 AM, Timothy Chen <tnachen@gmail.com> wrote:
>>
>>> Ok I created SPARK-3817 to track this, will try to repro it as well.
>>>
>>> Tim
>>>
>>> On Mon, Oct 6, 2014 at 6:08 AM, RJ Nowling <rnowling@gmail.com> wrote:
>>> > I've recently run into this issue as well. I get it from running Spark
>>> > examples such as log query.  Maybe that'll help reproduce the issue.
>>> >
>>> >
>>> > On Monday, October 6, 2014, Gurvinder Singh <
>>> gurvinder.singh@uninett.no>
>>> > wrote:
>>> >>
>>> >> The issue does not occur if the task at hand has small number of map
>>> >> tasks. I have a task which has 978 map tasks and I see this error as
>>> >>
>>> >> 14/10/06 09:34:40 ERROR BlockManagerMasterActor: Got two different
>>> block
>>> >> manager registrations on 20140711-081617-711206558-5050-2543-5
>>> >>
>>> >> Here is the log from the mesos-slave where this container was running.
>>> >>
>>> >> http://pastebin.com/Q1Cuzm6Q
>>> >>
>>> >> If you look for the code from where error produced by spark, you will
>>> >> see that it simply exit and saying in comments "this should never
>>> >> happen, lets just quit" :-)
>>> >>
>>> >> - Gurvinder
>>> >> On 10/06/2014 09:30 AM, Timothy Chen wrote:
>>> >> > (Hit enter too soon...)
>>> >> >
>>> >> > What is your setup and steps to repro this?
>>> >> >
>>> >> > Tim
>>> >> >
>>> >> > On Mon, Oct 6, 2014 at 12:30 AM, Timothy Chen <tnachen@gmail.com>
>>> wrote:
>>> >> >> Hi Gurvinder,
>>> >> >>
>>> >> >> I tried fine grain mode before and didn't get into that problem.
>>> >> >>
>>> >> >>
>>> >> >> On Sun, Oct 5, 2014 at 11:44 PM, Gurvinder Singh
>>> >> >> <gurvinder.singh@uninett.no> wrote:
>>> >> >>> On 10/06/2014 08:19 AM, Fairiz Azizi wrote:
>>> >> >>>> The Spark online docs indicate that Spark is compatible
with
>>> Mesos
>>> >> >>>> 0.18.1
>>> >> >>>>
>>> >> >>>> I've gotten it to work just fine on 0.18.1 and 0.18.2
>>> >> >>>>
>>> >> >>>> Has anyone tried Spark on a newer version of Mesos,
i.e. Mesos
>>> >> >>>> v0.20.0?
>>> >> >>>>
>>> >> >>>> -Fi
>>> >> >>>>
>>> >> >>> Yeah we are using Spark 1.1.0 with Mesos 0.20.1. It runs
fine in
>>> >> >>> coarse
>>> >> >>> mode, in fine grain mode there is an issue with blockmanager
names
>>> >> >>> conflict. I have been waiting for it to be fixed but it
is still
>>> >> >>> there.
>>> >> >>>
>>> >> >>> -Gurvinder
>>> >> >>>
>>> >> >>>
>>> ---------------------------------------------------------------------
>>> >> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> >> >>> For additional commands, e-mail: dev-help@spark.apache.org
>>> >> >>>
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> >> For additional commands, e-mail: dev-help@spark.apache.org
>>> >>
>>> >
>>> >
>>> > --
>>> > em rnowling@gmail.com
>>> > c 954.496.2314
>>>
>>
>>
>
>
> --
> em rnowling@gmail.com
> c 954.496.2314
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message