spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Soni <asoni.le...@gmail.com>
Subject Re: Spark 1.5 on Mesos
Date Mon, 29 Feb 2016 18:23:19 GMT
What is the Best practice , I have everything running as docker container
in single host ( mesos and marathon also as docker container )  and
everything comes up fine but when i try to launch the spark shell i get
below error


SQL context available as sqlContext.

scala> val data = sc.parallelize(1 to 100)
data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at
parallelize at <console>:27

scala> data.count
[Stage 0:>                                                          (0 + 0)
/ 2]16/02/29 18:21:12 WARN TaskSchedulerImpl: Initial job has not accepted
any resources; check your cluster UI to ensure that workers are registered
and have sufficient resources
16/02/29 18:21:27 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient resources



On Mon, Feb 29, 2016 at 12:04 PM, Tim Chen <tim@mesosphere.io> wrote:

> No you don't have to run Mesos in docker containers to run Spark in docker
> containers.
>
> Once you have Mesos cluster running you can then specfiy the Spark
> configurations in your Spark job (i.e: spark.mesos.executor.docker.image=mesosphere/spark:1.6)
> and Mesos will automatically launch docker containers for you.
>
> Tim
>
> On Mon, Feb 29, 2016 at 7:36 AM, Ashish Soni <asoni.learn@gmail.com>
> wrote:
>
>> Yes i read that and not much details here.
>>
>> Is it true that we need to have spark installed on each mesos docker
>> container ( master and slave ) ...
>>
>> Ashish
>>
>> On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen <tim@mesosphere.io> wrote:
>>
>>> https://spark.apache.org/docs/latest/running-on-mesos.html should be
>>> the best source, what problems were you running into?
>>>
>>> Tim
>>>
>>> On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang <yy201602@gmail.com> wrote:
>>>
>>>> Have you read this ?
>>>> https://spark.apache.org/docs/latest/running-on-mesos.html
>>>>
>>>> On Fri, Feb 26, 2016 at 11:03 AM, Ashish Soni <asoni.learn@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All ,
>>>>>
>>>>> Is there any proper documentation as how to run spark on mesos , I am
>>>>> trying from the last few days and not able to make it work.
>>>>>
>>>>> Please help
>>>>>
>>>>> Ashish
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message