spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <shushantaror...@gmail.com>
Subject Re: spark on yarn
Date Tue, 14 Jul 2015 17:40:21 GMT
got the below exception in logs:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):
Invalid resource request, requested virtual cores < 0, or requested virtual
cores > max configured, requestedVirtualCores=5, maxVirtualCores=4
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:205)
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:94)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:487)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)


-----------------------------------------------------------------------------------------------------------------------------------
My understanding was --executor-cores(5 here) are maximum concurrent tasks
possible in an executor and --num-executors (10 here)are no of executors or
containers demanded by Application master/Spark driver program  to yarn RM.

So these --num-executors (5) are parallel tasks and why is it controlled by
yarn . Why not by JVM executor which was started by name node . Why can't
these 10 JVM executors started by respective Name  nodes as containers
 allocate requested tasks or threads > vcores configured on node.

Is there a way I can control maxVirtualCores for an application and be it
very large than no of cores. Say 100 for a 8 core system. Since in a JVM
process( CPU+IO+Network intensive processing)100 threads are very less ?






On Tue, Jul 14, 2015 at 10:52 PM, Marcelo Vanzin <vanzin@cloudera.com>
wrote:

>
> On Tue, Jul 14, 2015 at 9:57 AM, Shushant Arora <shushantarora09@gmail.com
> > wrote:
>
>> When I specify --executor-cores > 4 it fails to start the application.
>> When I give --executor-cores as 4 , it works fine.
>>
>
> Do you have any NM that advertises more than 4 available cores?
>
> Also, it's always worth it to check if there's anything interesting in the
> logs; see the "yarn logs" command, and also the RM/NM logs.
>
> --
> Marcelo
>

Mime
View raw message