tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Dai <da...@hortonworks.com>
Subject Re: meet a problem when using "pig -x tez_local"
Date Fri, 06 Jun 2014 18:19:29 GMT
There are several fix for Order by in PIG-3846. I will try to commit
it soon and hope it solves the problem. Can your focus on
non-orderby/skewedjoin queries for now?

Thanks,
Daniel

On Fri, Jun 6, 2014 at 7:12 AM, Chen He <airbots@gmail.com> wrote:
> Hi Daniel
>
> I saw there is a line using "ordered by" in the script1-local.pig file.
>
> "ordered_uniq_frequency = ORDER filtered_uniq_frequency BY hour, score;"
>
> Regards!
>
> Chen
>
>
> On Fri, Jun 6, 2014 at 12:17 AM, Chen He <airbots@gmail.com> wrote:
>
>> Thank you for the reply, Daniel. I will check tomorrow. Actually, i just
>> use tutorial script1-local.
>>
>> Regards!
>>
>> Chen
>> On Jun 5, 2014 11:42 PM, "Daniel Dai" <daijy@hortonworks.com> wrote:
>>
>>> Hi, Chen,
>>> Are you seeing this issue only when running a query containing "order
>>> by"? It seems a Pig bug on this side, I can take a look tomorrow. But
>>> in worst case, we can disable ObjectRegistry in local mode.
>>>
>>> Thanks,
>>> Daniel
>>>
>>> On Thu, Jun 5, 2014 at 9:14 PM, Chen He <airbots@gmail.com> wrote:
>>> > Hi Hitesh
>>> >
>>> > ObjectRegistry is used for broadcastShuffle. If a vertex has more than
>>> one
>>> > task. Pig uses ObjectRegistry to save objects that can be used for all
>>> > tasks in a vertex.  The ObjectRegistryImpl's annotation is singleton. I
>>> > suspect it is a race condition between constructor and caller.
>>> >
>>> > Regards!
>>> >
>>> > Chen
>>> >
>>> >
>>> > On Thu, Jun 5, 2014 at 8:49 PM, Hitesh Shah <hitesh@apache.org> wrote:
>>> >
>>> >> @Chen, I am not sure asking for the user code to change is an option.
>>> We
>>> >> should address the underlying issue which is fixing ObjectRegistry to
>>> work
>>> >> in local mode.
>>> >>
>>> >> thanks
>>> >> -- Hitesh
>>> >>
>>> >>
>>> >> On Thu, Jun 5, 2014 at 1:34 PM, Chen He <airbots@gmail.com> wrote:
>>> >>
>>> >> > Hi Hitesh
>>> >> >
>>> >> > In my code, I did not touch any ObjectRegistry related code. I
>>> suspect
>>> >> the
>>> >> > instantiation process of ObjectRegistry  has problem so that it
>>> failed to
>>> >> > create an instance when it is called.
>>> >> >
>>> >> > I didn't know why PigProcessor collectSample vertex. But looks
like
>>> it is
>>> >> > not a must for TezLocalMode.
>>> >> >
>>> >> > Chen
>>> >> >
>>> >> >
>>> >> > On Thu, Jun 5, 2014 at 2:59 PM, Chen He <airbots@gmail.com>
wrote:
>>> >> >
>>> >> > > Hi Hitesh
>>> >> > >
>>> >> > > Yes, objectRegistry returns NPE. I saw it is a inject static
>>> variable.
>>> >> > >
>>> >> > > Regards!
>>> >> > >
>>> >> > > Chen
>>> >> > >
>>> >> > >
>>> >> > > On Thu, Jun 5, 2014 at 9:15 AM, Hitesh Shah <hitesh@apache.org>
>>> wrote:
>>> >> > >
>>> >> > >> @Chen, this might be a bug in the tez local execution
code. Can
>>> you
>>> >> > >> confirm
>>> >> > >> that ObjectRegistryFactory::getObjectRegistry returns
a non-null
>>> >> value?
>>> >> > >>
>>> >> > >> thanks
>>> >> > >> — Hitesh
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Jun 4, 2014 at 4:21 PM, Chen He <airbots@gmail.com>
>>> wrote:
>>> >> > >>
>>> >> > >> > I am using pig -x tez_local to test tez local mode
and met
>>> following
>>> >> > >> > problems (use tutorial.jar)
>>> >> > >> >
>>> >> > >> > 2014-06-03 11:06:35,964 [pool-1-thread-1] INFO
>>> >> > >> > org.apache.hadoop.mapred.YarnTezDagChild - Running
task,
>>> >> > >> > taskAttemptId=attempt_1401811588093_0000_1_05_000000_0
>>> >> > >> > 2014-06-03 11:06:35,965 [pool-1-thread-1] ERROR
>>> >> > >> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor
-
>>> >> > >> > Encountered exception while processing:
>>> >> > >> > java.lang.NullPointerException
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.tez.ObjectCache.retrieve(ObjectCache.java:47)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.collectSample(PigProcessor.java:307)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.initializeInputs(PigProcessor.java:218)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:162)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:317)
>>> >> > >> >         at
>>> >> > >> >
>>> >> >
>>> org.apache.hadoop.mapred.YarnTezDagChild$3.run(YarnTezDagChild.java:602)
>>> >> > >> >         at java.security.AccessController.doPrivileged(Native
>>> >> Method)
>>> >> > >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.hadoop.mapred.YarnTezDagChild.pollTask(YarnTezDagChild.java:591)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:332)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:250)
>>> >> > >> >         at java.lang.Thread.run(Thread.java:744)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >> > >> >         at
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >> > >> >         at java.lang.Thread.run(Thread.java:744)
>>> >> > >> >
>>> >> > >> > code checking implies that the ObjectCache is trying
to get
>>> >> something
>>> >> > >> from
>>> >> > >> > a Map but it is not there.
>>> >> > >> > But what is the purpose of using ObjectCache in PIG
on TEZ?
>>> >> > >> > Why pig needs sampleVertex in the PigProcessor?
>>> >> > >> >
>>> >> > >> > Any reply will be appreciated.
>>> >> > >> >
>>> >> > >> > Regards!
>>> >> > >> >
>>> >> > >> > Chen
>>> >> > >> >
>>> >> > >>
>>> >> > >
>>> >> > >
>>> >> >
>>> >>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>>
>>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message