tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Dai <da...@hortonworks.com>
Subject Re: meet a problem when using "pig -x tez_local"
Date Fri, 06 Jun 2014 04:41:35 GMT
Hi, Chen,
Are you seeing this issue only when running a query containing "order
by"? It seems a Pig bug on this side, I can take a look tomorrow. But
in worst case, we can disable ObjectRegistry in local mode.

Thanks,
Daniel

On Thu, Jun 5, 2014 at 9:14 PM, Chen He <airbots@gmail.com> wrote:
> Hi Hitesh
>
> ObjectRegistry is used for broadcastShuffle. If a vertex has more than one
> task. Pig uses ObjectRegistry to save objects that can be used for all
> tasks in a vertex.  The ObjectRegistryImpl's annotation is singleton. I
> suspect it is a race condition between constructor and caller.
>
> Regards!
>
> Chen
>
>
> On Thu, Jun 5, 2014 at 8:49 PM, Hitesh Shah <hitesh@apache.org> wrote:
>
>> @Chen, I am not sure asking for the user code to change is an option. We
>> should address the underlying issue which is fixing ObjectRegistry to work
>> in local mode.
>>
>> thanks
>> -- Hitesh
>>
>>
>> On Thu, Jun 5, 2014 at 1:34 PM, Chen He <airbots@gmail.com> wrote:
>>
>> > Hi Hitesh
>> >
>> > In my code, I did not touch any ObjectRegistry related code. I suspect
>> the
>> > instantiation process of ObjectRegistry  has problem so that it failed to
>> > create an instance when it is called.
>> >
>> > I didn't know why PigProcessor collectSample vertex. But looks like it is
>> > not a must for TezLocalMode.
>> >
>> > Chen
>> >
>> >
>> > On Thu, Jun 5, 2014 at 2:59 PM, Chen He <airbots@gmail.com> wrote:
>> >
>> > > Hi Hitesh
>> > >
>> > > Yes, objectRegistry returns NPE. I saw it is a inject static variable.
>> > >
>> > > Regards!
>> > >
>> > > Chen
>> > >
>> > >
>> > > On Thu, Jun 5, 2014 at 9:15 AM, Hitesh Shah <hitesh@apache.org> wrote:
>> > >
>> > >> @Chen, this might be a bug in the tez local execution code. Can you
>> > >> confirm
>> > >> that ObjectRegistryFactory::getObjectRegistry returns a non-null
>> value?
>> > >>
>> > >> thanks
>> > >> — Hitesh
>> > >>
>> > >>
>> > >> On Wed, Jun 4, 2014 at 4:21 PM, Chen He <airbots@gmail.com> wrote:
>> > >>
>> > >> > I am using pig -x tez_local to test tez local mode and met following
>> > >> > problems (use tutorial.jar)
>> > >> >
>> > >> > 2014-06-03 11:06:35,964 [pool-1-thread-1] INFO
>> > >> > org.apache.hadoop.mapred.YarnTezDagChild - Running task,
>> > >> > taskAttemptId=attempt_1401811588093_0000_1_05_000000_0
>> > >> > 2014-06-03 11:06:35,965 [pool-1-thread-1] ERROR
>> > >> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor
-
>> > >> > Encountered exception while processing:
>> > >> > java.lang.NullPointerException
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.ObjectCache.retrieve(ObjectCache.java:47)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.collectSample(PigProcessor.java:307)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.initializeInputs(PigProcessor.java:218)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:162)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:317)
>> > >> >         at
>> > >> >
>> > org.apache.hadoop.mapred.YarnTezDagChild$3.run(YarnTezDagChild.java:602)
>> > >> >         at java.security.AccessController.doPrivileged(Native
>> Method)
>> > >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>> > >> >         at
>> > >> >
>> > >>
>> >
>> org.apache.hadoop.mapred.YarnTezDagChild.pollTask(YarnTezDagChild.java:591)
>> > >> >         at
>> > >> >
>> > >>
>> >
>> org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:332)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:250)
>> > >> >         at java.lang.Thread.run(Thread.java:744)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > >> >         at
>> > >> >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > >> >         at java.lang.Thread.run(Thread.java:744)
>> > >> >
>> > >> > code checking implies that the ObjectCache is trying to get
>> something
>> > >> from
>> > >> > a Map but it is not there.
>> > >> > But what is the purpose of using ObjectCache in PIG on TEZ?
>> > >> > Why pig needs sampleVertex in the PigProcessor?
>> > >> >
>> > >> > Any reply will be appreciated.
>> > >> >
>> > >> > Regards!
>> > >> >
>> > >> > Chen
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message