hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sreejith Ramakrishnan <sreejith.c...@gmail.com>
Subject Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Date Thu, 11 Jul 2013 11:58:06 GMT
@Devaraj K

I tried setting mapred.jobtracker.maxtasks.per.job to 4 in mapred-site.xml.
In the subsequent job, I got this in the log which clearly shows the
configuration has taken effect:

INFO org.apache.hadoop.mapred.JobInProgress: job_201307111721_0001: nMaps=1
nReduces=1 max=4

Yet, the original problem remains. It doesn't go anywhere from 0%. Btw,
when we get the jobQueue, should we specify a name for the queue we want?
(eg. default)

INFO org.apache.hadoop.mapred.JobTracker: Job job_201307111721_0001 added
successfully for user 'sreejith' to queue 'default'

Also, is there a limit to the no.of tasks which can be obtained in a
heartbeat?


On Thu, Jul 11, 2013 at 2:15 PM, Devaraj k <devaraj.k@huawei.com> wrote:

> You don't need to explicitly call the initTasks(), you can find this
> invocation in JobTracker.java class.
>
> >*Yet the **Cannot create task split for <my-job-id>** is still there*.
> >About errors relating to the JobID, is this significant? --- INFO
> >org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> >nReduces=1 max=-1
>
> Here it shows max=-1, which means max configured(i.e default) tasks for
> the Job is -1.
>
> I think you can see this exception from JT log.
>
> LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks
> + " max=" + maxTasks);
>     if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
>       throw new IOException(
>                 "The number of tasks for this job " +
>                 (numMapTasks + numReduceTasks) +
>                 " exceeds the configured limit " + maxTasks);
>     }
>
> Can you try configuring some number for this configuration
> 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
>
>
> Thanks
> Devaraj k
>
>
> -----Original Message-----
> From: Sreejith Ramakrishnan [mailto:sreejith.code@gmail.com]
> Sent: 11 July 2013 13:26
> To: mapreduce-dev@hadoop.apache.org
> Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
>
> @Devaraj K
>
> *I tried explicitly calling incomingJob.initTasks()*and it gave me a
> pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
> job_201307111248_0002 initialized successfully with 1 map tasks and 1
> reduce tasks
>
> *Yet the **Cannot create task split for <my-job-id>** is still there*.
> About errors relating to the JobID, is this significant? --- INFO
> org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> nReduces=1 max=-1
>
> *P.S: In the log file I shared, anything with "sreejith == " were logged
> by me. Not system-generated*
>
>
> On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan <
> sreejith.code@gmail.com> wrote:
>
> > I've attached the relevant portion of the log.
> >
> > For easy testing, *I've attached the maven project* for the
> > EDFJobScheduler. Just make a JAR from it and put it in
> > $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
> >
> > *<property>
> > *
> > *    <name>mapred.jobtracker.taskScheduler</name>
> >     <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> > *
> > *</property>*
> >
> >
> > On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> > sreejith.code@gmail.com> wrote:
> >
> >> I'm making my own scheduler. Since I had errors, I rewrote the
> >> scheduler to do just a simple thing. When I give it a job, it
> >> allocates 1 map() and 1
> >> reduce() to it. But that isn't happening. I used a barebones WordCount
> pgm.
> >> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
> >>
> >> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split
> >> for <my-job-id>
> >>
> >> DETAILED INFO:
> >>
> >> Here's some snippets of code. The scheduler logic is inside the
> >> assignTasks():
> >>
> >> We get a JobInProgressListener in the constructor-
> >> this.jobQueueJobInProgressListener = new
> >> JobQueueJobInProgressListener();
> >>
> >> In start() -
> >>
> >> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
> >> sListener);
> >>
> >> Then I just take the last element in that queue (I assume its the
> >> latest submitted job) and assign to incomingJob (of type JobInProgress).
> >>
> >> Then I obtain a new map task as follows and append this task to a
> >> list which is the return value of the assignTasks():
> >> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
> >> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
> >>
> >>
> >> I've attached the scheduler. Can any expert point out if there's any
> >> ignorant mistakes from my side?
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message