tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen He <airb...@gmail.com>
Subject Re: meet a problem when using "pig -x tez_local"
Date Fri, 06 Jun 2014 04:14:30 GMT
Hi Hitesh

ObjectRegistry is used for broadcastShuffle. If a vertex has more than one
task. Pig uses ObjectRegistry to save objects that can be used for all
tasks in a vertex.  The ObjectRegistryImpl's annotation is singleton. I
suspect it is a race condition between constructor and caller.

Regards!

Chen


On Thu, Jun 5, 2014 at 8:49 PM, Hitesh Shah <hitesh@apache.org> wrote:

> @Chen, I am not sure asking for the user code to change is an option. We
> should address the underlying issue which is fixing ObjectRegistry to work
> in local mode.
>
> thanks
> -- Hitesh
>
>
> On Thu, Jun 5, 2014 at 1:34 PM, Chen He <airbots@gmail.com> wrote:
>
> > Hi Hitesh
> >
> > In my code, I did not touch any ObjectRegistry related code. I suspect
> the
> > instantiation process of ObjectRegistry  has problem so that it failed to
> > create an instance when it is called.
> >
> > I didn't know why PigProcessor collectSample vertex. But looks like it is
> > not a must for TezLocalMode.
> >
> > Chen
> >
> >
> > On Thu, Jun 5, 2014 at 2:59 PM, Chen He <airbots@gmail.com> wrote:
> >
> > > Hi Hitesh
> > >
> > > Yes, objectRegistry returns NPE. I saw it is a inject static variable.
> > >
> > > Regards!
> > >
> > > Chen
> > >
> > >
> > > On Thu, Jun 5, 2014 at 9:15 AM, Hitesh Shah <hitesh@apache.org> wrote:
> > >
> > >> @Chen, this might be a bug in the tez local execution code. Can you
> > >> confirm
> > >> that ObjectRegistryFactory::getObjectRegistry returns a non-null
> value?
> > >>
> > >> thanks
> > >> — Hitesh
> > >>
> > >>
> > >> On Wed, Jun 4, 2014 at 4:21 PM, Chen He <airbots@gmail.com> wrote:
> > >>
> > >> > I am using pig -x tez_local to test tez local mode and met following
> > >> > problems (use tutorial.jar)
> > >> >
> > >> > 2014-06-03 11:06:35,964 [pool-1-thread-1] INFO
> > >> > org.apache.hadoop.mapred.YarnTezDagChild - Running task,
> > >> > taskAttemptId=attempt_1401811588093_0000_1_05_000000_0
> > >> > 2014-06-03 11:06:35,965 [pool-1-thread-1] ERROR
> > >> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor -
> > >> > Encountered exception while processing:
> > >> > java.lang.NullPointerException
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> org.apache.pig.backend.hadoop.executionengine.tez.ObjectCache.retrieve(ObjectCache.java:47)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.collectSample(PigProcessor.java:307)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.initializeInputs(PigProcessor.java:218)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:162)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:317)
> > >> >         at
> > >> >
> > org.apache.hadoop.mapred.YarnTezDagChild$3.run(YarnTezDagChild.java:602)
> > >> >         at java.security.AccessController.doPrivileged(Native
> Method)
> > >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> > >> >         at
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.YarnTezDagChild.pollTask(YarnTezDagChild.java:591)
> > >> >         at
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:332)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:250)
> > >> >         at java.lang.Thread.run(Thread.java:744)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > >> >         at
> > >> >
> > >> >
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >> >         at java.lang.Thread.run(Thread.java:744)
> > >> >
> > >> > code checking implies that the ObjectCache is trying to get
> something
> > >> from
> > >> > a Map but it is not there.
> > >> > But what is the purpose of using ObjectCache in PIG on TEZ?
> > >> > Why pig needs sampleVertex in the PigProcessor?
> > >> >
> > >> > Any reply will be appreciated.
> > >> >
> > >> > Regards!
> > >> >
> > >> > Chen
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message