Have you looked at the cluster UI, and do you see any workers registered there, and your application under running applications? Maybe you typed in the wrong master URL or something like that.

Matei

On Jan 8, 2014, at 7:07 PM, Aureliano Buendia <buendia360@gmail.com> wrote:

The strange thing is that spark examples work fine, but when I include a spark example in my jar and deploy it, I get this error for the very same example:

WARN ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

My jar is deployed to master and then to workers by spark-ec2/copy-dir. Why would including the example in my jar cause this error?



On Thu, Jan 9, 2014 at 12:41 AM, Aureliano Buendia <buendia360@gmail.com> wrote:
Could someone explain how SPARK_MEM, SPARK_WORKER_MEMORY and spark.executor.memory should be related so that this non helpful error doesn't occur?

Maybe there are more env and java config variable about memory that I'm missing.

By the way, that bit of the error asking to check the web UI, it's just redundant. The UI is of no help.


On Wed, Jan 8, 2014 at 4:31 PM, Aureliano Buendia <buendia360@gmail.com> wrote:
Hi,


My spark cluster is not able to run a job due to this warning:

WARN ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

The workers have these status:

ALIVE 2 (0 Used)6.3 GB (0.0 B Used)
So there must be plenty of memory available despite the warning message. I'm using default spark config, is there a config parameter that needs changing for this to work?