tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen He <airb...@gmail.com>
Subject Re: Pig on tez errors
Date Tue, 27 May 2014 16:04:07 GMT
Hi Daniel

Thank you for your reply. Any suggestion about how can I run pig local mode
using Tez without minicluster? I am sure the code on Tez side is ready.
What changes need to be made on PIG? I am not very familar with PIG.

Regards!

Chen


On Mon, May 26, 2014 at 8:22 PM, Chen He <airbots@gmail.com> wrote:

> Hi Jon, Daniel
>
> Daniel was right. I tried pig local mode using "pig -x local -x tez …..".
> It tries to connect mini cluster and finally timeout. The error that I
> mentioned before : "ERROR 2998: Unhandled internal error. Found interface
> org.apache.hadoop.yarn.api.records.LocalResource, but class was expected"
> is because I have a hadoop 0.23 mini cluster running in background.
>
> Chen
>
>
> On Mon, May 26, 2014 at 9:39 AM, Jonathan Eagles <jeagles@gmail.com>wrote:
>
>> Hi Daniel,
>>
>> You've been very helpful in your replies. I believe the missing context is
>> that Chen is working on adding Tez Local Mode (TEZ-684) and is trying to
>> integrate with Pig. The question as I interpret it is whether Pig needs or
>> creates a MiniCluster in local mode. Chen can you confirm this?
>>
>> Jon
>>
>>
>> On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <daijy@hortonworks.com>
>> wrote:
>>
>> > Hi, Chen,
>> >
>> > Please note Pig local mode does not run tez. It uses MR engine.
>> > Tez-local mode is not implemented (depend on TEZ-235). You will need a
>> > real cluster or use TezMiniCluster to test Pig on Tez.
>> >
>> > Thanks,
>> > Daniel
>> >
>> > On Sun, May 25, 2014 at 11:42 AM, Chen He <airbots@gmail.com> wrote:
>> > > Hi Daniel
>> > >
>> > > I was testing the pig local mode. Does pig need mini cluster when it
>> runs
>> > > in local mode?
>> > >
>> > > Regards!
>> > >
>> > > Chen
>> > >
>> > >
>> > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai <daijy@hortonworks.com>
>> > wrote:
>> > >
>> > >> Which Hadoop are you running against? Also 2.2?
>> > >>
>> > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <airbots@gmail.com>
wrote:
>> > >> > Hi Daniel
>> > >> >
>> > >> > Thank you for the reply. I compiled Tez and Pig against hadoop
2.2.
>> > >> >
>> > >> > Regards!
>> > >> >
>> > >> > Chen
>> > >> >
>> > >> >
>> > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai <daijy@hortonworks.com
>> >
>> > >> wrote:
>> > >> >
>> > >> >> Which version of Hadoop are you using? Seem LocalResource
changed
>> > from
>> > >> >> interface to class in 2.1 (YARN-711). If you are using pre-2.1,
>> this
>> > >> >> might be the case.
>> > >> >>
>> > >> >> Thanks,
>> > >> >> Daniel
>> > >> >>
>> > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <airbots@gmail.com>
>> wrote:
>> > >> >> > Hi Daniel
>> > >> >> >
>> > >> >> > Thank you for the reply. I did as you suggested but get
>> following
>> > >> errors:
>> > >> >> >
>> > >> >> > ERROR 2998: Unhandled internal error. Found interface
>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but
class was
>> > >> expected
>> > >> >> >
>> > >> >> > java.lang.IncompatibleClassChangeError: Found interface
>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but
class was
>> > >> expected
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
>> > >> >> >     at org.apache.pig.PigServer.launchPlan(PigServer.java:1378)
>> > >> >> >     at
>> > >> >> >
>> > >>
>> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363)
>> > >> >> >     at org.apache.pig.PigServer.execute(PigServer.java:1352)
>> > >> >> >     at org.apache.pig.PigServer.executeBatch(PigServer.java:403)
>> > >> >> >     at org.apache.pig.PigServer.executeBatch(PigServer.java:386)
>> > >> >> >     at
>> > >> >> >
>> > >>
>> >
>> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
>> > >> >> >     at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
>> > >> >> >     at org.apache.pig.Main.run(Main.java:608)
>> > >> >> >     at org.apache.pig.Main.main(Main.java:156)
>> > >> >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > >> >> >     at
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > >> >> >     at java.lang.reflect.Method.invoke(Method.java:606)
>> > >> >> >     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> > >> >> >
>> > >> >> > Is this a problem that caused by hadoop incompatiblilty?
Which
>> > >> version of
>> > >> >> > Hadoop is the best?
>> > >> >> >
>> > >> >> > Regards!
>> > >> >> >
>> > >> >> > Chen
>> > >> >> >
>> > >> >> >
>> > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel Dai <
>> daijy@hortonworks.com
>> > >
>> > >> >> wrote:
>> > >> >> >
>> > >> >> >> Need to set PIG_CLASSPATH:
>> > >> >> >> export PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf
>> > >> >> >>
>> > >> >> >> Thanks,
>> > >> >> >> Daniel
>> > >> >> >>
>> > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen He <airbots@gmail.com>
>> > wrote:
>> > >> >> >> > Any one met this error before?
>> > >> >> >> >
>> > >> >> >> > [main] ERROR org.apache.pig.Main - ERROR 2998:
Unhandled
>> > internal
>> > >> >> error.
>> > >> >> >> > org/apache/tez/common/counters/DAGCounter
>> > >> >> >> >
>> > >> >> >> > Any reply will be appreciated!
>> > >> >> >> >
>> > >> >> >> > Regards!
>> > >> >> >> >
>> > >> >> >> > Chen
>> > >> >> >>
>> > >> >> >> --
>> > >> >> >> CONFIDENTIALITY NOTICE
>> > >> >> >> NOTICE: This message is intended for the use of the
individual
>> or
>> > >> >> entity to
>> > >> >> >> which it is addressed and may contain information
that is
>> > >> confidential,
>> > >> >> >> privileged and exempt from disclosure under applicable
law. If
>> the
>> > >> >> reader
>> > >> >> >> of this message is not the intended recipient, you
are hereby
>> > >> notified
>> > >> >> that
>> > >> >> >> any printing, copying, dissemination, distribution,
disclosure
>> or
>> > >> >> >> forwarding of this communication is strictly prohibited.
If you
>> > have
>> > >> >> >> received this communication in error, please contact
the sender
>> > >> >> immediately
>> > >> >> >> and delete it from your system. Thank You.
>> > >> >> >>
>> > >> >>
>> > >> >> --
>> > >> >> CONFIDENTIALITY NOTICE
>> > >> >> NOTICE: This message is intended for the use of the individual
or
>> > >> entity to
>> > >> >> which it is addressed and may contain information that is
>> > confidential,
>> > >> >> privileged and exempt from disclosure under applicable law.
If the
>> > >> reader
>> > >> >> of this message is not the intended recipient, you are hereby
>> > notified
>> > >> that
>> > >> >> any printing, copying, dissemination, distribution, disclosure
or
>> > >> >> forwarding of this communication is strictly prohibited. If
you
>> have
>> > >> >> received this communication in error, please contact the sender
>> > >> immediately
>> > >> >> and delete it from your system. Thank You.
>> > >> >>
>> > >>
>> > >> --
>> > >> CONFIDENTIALITY NOTICE
>> > >> NOTICE: This message is intended for the use of the individual or
>> > entity to
>> > >> which it is addressed and may contain information that is
>> confidential,
>> > >> privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > >> of this message is not the intended recipient, you are hereby
>> notified
>> > that
>> > >> any printing, copying, dissemination, distribution, disclosure or
>> > >> forwarding of this communication is strictly prohibited. If you have
>> > >> received this communication in error, please contact the sender
>> > immediately
>> > >> and delete it from your system. Thank You.
>> > >>
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message