spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: multiple pyspark instances simultaneously (same time)
Date Thu, 22 Oct 2015 11:40:12 GMT
Did you read
https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application

Thanks
Best Regards

On Thu, Oct 15, 2015 at 11:31 PM, jeff.sadowski@gmail.com <
jeff.sadowski@gmail.com> wrote:

> I am having issues trying to setup spark to run jobs simultaneously.
>
> I thought I wanted FAIR scheduling?
>
> I used the templated fairscheduler.xml as is when I start pyspark I see the
> 3 expected pools:
> production, test, and default
>
> when I login as second user and run pyspark
> I see the expected pools as that user as well
>
> when I open up a webbrowser to http://master:8080
>
> I see my first user's state is running and my second user's state is
> waiting
>
> so I try putting them both in the production pool which is fair scheduler
>
> When I refresh http://master:8080
>
> the second user's status is still waiting.
>
> If I try to run something as the second user I get
>
> "Initial job has not accepted any resources"
>
> Maybe fair queuing is not what I want?
>
> I'm starting pyspark as follows
>
> pyspark --master spark://master:7077
>
> I started spark as follows
>
> start-all.sh
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/multiple-pyspark-instances-simultaneously-same-time-tp25079.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message