tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen He <airb...@gmail.com>
Subject Re: meet a problem when using "pig -x tez_local"
Date Fri, 06 Jun 2014 14:12:48 GMT
Hi Daniel

I saw there is a line using "ordered by" in the script1-local.pig file.

"ordered_uniq_frequency = ORDER filtered_uniq_frequency BY hour, score;"

Regards!

Chen


On Fri, Jun 6, 2014 at 12:17 AM, Chen He <airbots@gmail.com> wrote:

> Thank you for the reply, Daniel. I will check tomorrow. Actually, i just
> use tutorial script1-local.
>
> Regards!
>
> Chen
> On Jun 5, 2014 11:42 PM, "Daniel Dai" <daijy@hortonworks.com> wrote:
>
>> Hi, Chen,
>> Are you seeing this issue only when running a query containing "order
>> by"? It seems a Pig bug on this side, I can take a look tomorrow. But
>> in worst case, we can disable ObjectRegistry in local mode.
>>
>> Thanks,
>> Daniel
>>
>> On Thu, Jun 5, 2014 at 9:14 PM, Chen He <airbots@gmail.com> wrote:
>> > Hi Hitesh
>> >
>> > ObjectRegistry is used for broadcastShuffle. If a vertex has more than
>> one
>> > task. Pig uses ObjectRegistry to save objects that can be used for all
>> > tasks in a vertex.  The ObjectRegistryImpl's annotation is singleton. I
>> > suspect it is a race condition between constructor and caller.
>> >
>> > Regards!
>> >
>> > Chen
>> >
>> >
>> > On Thu, Jun 5, 2014 at 8:49 PM, Hitesh Shah <hitesh@apache.org> wrote:
>> >
>> >> @Chen, I am not sure asking for the user code to change is an option.
>> We
>> >> should address the underlying issue which is fixing ObjectRegistry to
>> work
>> >> in local mode.
>> >>
>> >> thanks
>> >> -- Hitesh
>> >>
>> >>
>> >> On Thu, Jun 5, 2014 at 1:34 PM, Chen He <airbots@gmail.com> wrote:
>> >>
>> >> > Hi Hitesh
>> >> >
>> >> > In my code, I did not touch any ObjectRegistry related code. I
>> suspect
>> >> the
>> >> > instantiation process of ObjectRegistry  has problem so that it
>> failed to
>> >> > create an instance when it is called.
>> >> >
>> >> > I didn't know why PigProcessor collectSample vertex. But looks like
>> it is
>> >> > not a must for TezLocalMode.
>> >> >
>> >> > Chen
>> >> >
>> >> >
>> >> > On Thu, Jun 5, 2014 at 2:59 PM, Chen He <airbots@gmail.com> wrote:
>> >> >
>> >> > > Hi Hitesh
>> >> > >
>> >> > > Yes, objectRegistry returns NPE. I saw it is a inject static
>> variable.
>> >> > >
>> >> > > Regards!
>> >> > >
>> >> > > Chen
>> >> > >
>> >> > >
>> >> > > On Thu, Jun 5, 2014 at 9:15 AM, Hitesh Shah <hitesh@apache.org>
>> wrote:
>> >> > >
>> >> > >> @Chen, this might be a bug in the tez local execution code.
Can
>> you
>> >> > >> confirm
>> >> > >> that ObjectRegistryFactory::getObjectRegistry returns a non-null
>> >> value?
>> >> > >>
>> >> > >> thanks
>> >> > >> — Hitesh
>> >> > >>
>> >> > >>
>> >> > >> On Wed, Jun 4, 2014 at 4:21 PM, Chen He <airbots@gmail.com>
>> wrote:
>> >> > >>
>> >> > >> > I am using pig -x tez_local to test tez local mode and
met
>> following
>> >> > >> > problems (use tutorial.jar)
>> >> > >> >
>> >> > >> > 2014-06-03 11:06:35,964 [pool-1-thread-1] INFO
>> >> > >> > org.apache.hadoop.mapred.YarnTezDagChild - Running task,
>> >> > >> > taskAttemptId=attempt_1401811588093_0000_1_05_000000_0
>> >> > >> > 2014-06-03 11:06:35,965 [pool-1-thread-1] ERROR
>> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor
-
>> >> > >> > Encountered exception while processing:
>> >> > >> > java.lang.NullPointerException
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.tez.ObjectCache.retrieve(ObjectCache.java:47)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.collectSample(PigProcessor.java:307)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.initializeInputs(PigProcessor.java:218)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:162)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:317)
>> >> > >> >         at
>> >> > >> >
>> >> >
>> org.apache.hadoop.mapred.YarnTezDagChild$3.run(YarnTezDagChild.java:602)
>> >> > >> >         at java.security.AccessController.doPrivileged(Native
>> >> Method)
>> >> > >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>> >> > >> >         at
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.mapred.YarnTezDagChild.pollTask(YarnTezDagChild.java:591)
>> >> > >> >         at
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:332)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:250)
>> >> > >> >         at java.lang.Thread.run(Thread.java:744)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >> > >> >         at
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >> > >> >         at java.lang.Thread.run(Thread.java:744)
>> >> > >> >
>> >> > >> > code checking implies that the ObjectCache is trying
to get
>> >> something
>> >> > >> from
>> >> > >> > a Map but it is not there.
>> >> > >> > But what is the purpose of using ObjectCache in PIG on
TEZ?
>> >> > >> > Why pig needs sampleVertex in the PigProcessor?
>> >> > >> >
>> >> > >> > Any reply will be appreciated.
>> >> > >> >
>> >> > >> > Regards!
>> >> > >> >
>> >> > >> > Chen
>> >> > >> >
>> >> > >>
>> >> > >
>> >> > >
>> >> >
>> >>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message