tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen He <airb...@gmail.com>
Subject Re: Pig on tez errors
Date Tue, 27 May 2014 18:19:49 GMT
Hi Daniel

I was using this command: "/bin/pig -x local -x tez
tutorial/scripts/script1-local.pig"


On Tue, May 27, 2014 at 1:07 PM, Daniel Dai <daijy@hortonworks.com> wrote:

> Hi, Chen,
>
> The hook for for tez local mode is:
>
> bin/pig -x tez_local
>
> You might need to set PIG_OPTS="-Dopt.fetch=false" to prevent it
> bypassing the tez engine.
>
> How do you run tutorial example? Using MR local mode?
>
> Thanks,
> Daniel
>
> On Tue, May 27, 2014 at 9:32 AM, Chen He <airbots@gmail.com> wrote:
> > We are running pig tutorial example and get following errors. I think we
> > are pick up some hadoop config that specify HDFS instead of localFS.
> >
> > Here are the errors from pig.log
> >
> > Pig Stack Trace
> > ---------------
> > ERROR 6007: Unable to check name hdfs://localhost:9000/user/chenhe
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> > during parsing. Unable to check name hdfs://localhost:9000/user/chenhe
> >     at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1736)
> >     at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1431)
> >     at org.apache.pig.PigServer.parseAndBuild(PigServer.java:375)
> >     at org.apache.pig.PigServer.executeBatch(PigServer.java:400)
> >     at org.apache.pig.PigServer.executeBatch(PigServer.java:386)
> >     at
> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
> >     at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
> >     at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
> >     at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> >     at org.apache.pig.Main.run(Main.java:608)
> >     at org.apache.pig.Main.main(Main.java:156)
> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >     at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >     at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >     at java.lang.reflect.Method.invoke(Method.java:606)
> >     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > Caused by: Failed to parse: Pig script failed to parse:
> > <file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6>
> > pig script failed to validate:
> > org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007:
> Unable
> > to check name hdfs://localhost:9000/user/chenhe
> >     at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
> >     at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1723)
> >     ... 15 more
> > Caused by:
> > <file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6>
> > pig script failed to validate:
> > org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007:
> Unable
> > to check name hdfs://localhost:9000/user/chenhe
> >     at
> >
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:881)
> >     at
> >
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
> >     at
> >
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
> >     at
> >
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
> >     at
> >
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
> >     at
> >
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
> >     at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
> >     ... 16 more
> > Caused by: org.apache.pig.backend.datastorage.DataStorageException: ERROR
> > 6007: Unable to check name hdfs://localhost:9000/user/chenhe
> >     at
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:207)
> >     at
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
> >     at
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:138)
> >     at
> >
> org.apache.pig.parser.QueryParserUtils.getCurrentDir(QueryParserUtils.java:90)
> >     at
> >
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:873)
> >     ... 22 more
> > Caused by: java.net.ConnectException: Call From
> > guardwritten-dl.corp.yahoo.com/10.74.90.55 to localhost:9000 failed on
> > connection exception: java.net.ConnectException: Connection refused; For
> > more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
> >     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:700)
> >     at org.apache.hadoop.ipc.Client.call(Client.java:1098)
> >     at
> >
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
> >     at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source)
> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >     at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >     at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >     at java.lang.reflect.Method.invoke(Method.java:606)
> >     at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
> >     at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
> >     at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source)
> >     at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1305)
> >     at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734)
> >     at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272)
> >     at
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
> >     ... 26 more
> > Caused by: java.net.ConnectException: Connection refused
> >     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >     at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
> >     at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> >     at
> > org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:462)
> >     at
> > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:559)
> >     at
> org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:207)
> >     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1204)
> >     at org.apache.hadoop.ipc.Client.call(Client.java:1074)
> >     ... 39 more
> >
> ================================================================================
> >
> >
> >
> > On Tue, May 27, 2014 at 11:04 AM, Chen He <airbots@gmail.com> wrote:
> >
> >> Hi Daniel
> >>
> >> Thank you for your reply. Any suggestion about how can I run pig local
> >> mode using Tez without minicluster? I am sure the code on Tez side is
> >> ready. What changes need to be made on PIG? I am not very familar with
> PIG.
> >>
> >> Regards!
> >>
> >> Chen
> >>
> >>
> >> On Mon, May 26, 2014 at 8:22 PM, Chen He <airbots@gmail.com> wrote:
> >>
> >>> Hi Jon, Daniel
> >>>
> >>> Daniel was right. I tried pig local mode using "pig -x local -x tez
> …..".
> >>> It tries to connect mini cluster and finally timeout. The error that I
> >>> mentioned before : "ERROR 2998: Unhandled internal error. Found
> >>> interface org.apache.hadoop.yarn.api.records.LocalResource, but class
> >>> was expected"
> >>> is because I have a hadoop 0.23 mini cluster running in background.
> >>>
> >>> Chen
> >>>
> >>>
> >>> On Mon, May 26, 2014 at 9:39 AM, Jonathan Eagles <jeagles@gmail.com
> >wrote:
> >>>
> >>>> Hi Daniel,
> >>>>
> >>>> You've been very helpful in your replies. I believe the missing
> context
> >>>> is
> >>>> that Chen is working on adding Tez Local Mode (TEZ-684) and is trying
> to
> >>>> integrate with Pig. The question as I interpret it is whether Pig
> needs
> >>>> or
> >>>> creates a MiniCluster in local mode. Chen can you confirm this?
> >>>>
> >>>> Jon
> >>>>
> >>>>
> >>>> On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <daijy@hortonworks.com>
> >>>> wrote:
> >>>>
> >>>> > Hi, Chen,
> >>>> >
> >>>> > Please note Pig local mode does not run tez. It uses MR engine.
> >>>> > Tez-local mode is not implemented (depend on TEZ-235). You will
> need a
> >>>> > real cluster or use TezMiniCluster to test Pig on Tez.
> >>>> >
> >>>> > Thanks,
> >>>> > Daniel
> >>>> >
> >>>> > On Sun, May 25, 2014 at 11:42 AM, Chen He <airbots@gmail.com>
> wrote:
> >>>> > > Hi Daniel
> >>>> > >
> >>>> > > I was testing the pig local mode. Does pig need mini cluster
when
> it
> >>>> runs
> >>>> > > in local mode?
> >>>> > >
> >>>> > > Regards!
> >>>> > >
> >>>> > > Chen
> >>>> > >
> >>>> > >
> >>>> > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai <
> daijy@hortonworks.com>
> >>>> > wrote:
> >>>> > >
> >>>> > >> Which Hadoop are you running against? Also 2.2?
> >>>> > >>
> >>>> > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <airbots@gmail.com>
> wrote:
> >>>> > >> > Hi Daniel
> >>>> > >> >
> >>>> > >> > Thank you for the reply. I compiled Tez and Pig against
hadoop
> >>>> 2.2.
> >>>> > >> >
> >>>> > >> > Regards!
> >>>> > >> >
> >>>> > >> > Chen
> >>>> > >> >
> >>>> > >> >
> >>>> > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai <
> >>>> daijy@hortonworks.com>
> >>>> > >> wrote:
> >>>> > >> >
> >>>> > >> >> Which version of Hadoop are you using? Seem LocalResource
> changed
> >>>> > from
> >>>> > >> >> interface to class in 2.1 (YARN-711). If you
are using
> pre-2.1,
> >>>> this
> >>>> > >> >> might be the case.
> >>>> > >> >>
> >>>> > >> >> Thanks,
> >>>> > >> >> Daniel
> >>>> > >> >>
> >>>> > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <airbots@gmail.com>
> >>>> wrote:
> >>>> > >> >> > Hi Daniel
> >>>> > >> >> >
> >>>> > >> >> > Thank you for the reply. I did as you suggested
but get
> >>>> following
> >>>> > >> errors:
> >>>> > >> >> >
> >>>> > >> >> > ERROR 2998: Unhandled internal error. Found
interface
> >>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource,
but class
> was
> >>>> > >> expected
> >>>> > >> >> >
> >>>> > >> >> > java.lang.IncompatibleClassChangeError:
Found interface
> >>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource,
but class
> was
> >>>> > >> expected
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
> >>>> > >> >> >     at
> org.apache.pig.PigServer.launchPlan(PigServer.java:1378)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >>
> >>>>
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363)
> >>>> > >> >> >     at org.apache.pig.PigServer.execute(PigServer.java:1352)
> >>>> > >> >> >     at
> >>>> org.apache.pig.PigServer.executeBatch(PigServer.java:403)
> >>>> > >> >> >     at
> >>>> org.apache.pig.PigServer.executeBatch(PigServer.java:386)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
> >>>> > >> >> >     at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> >>>> > >> >> >     at org.apache.pig.Main.run(Main.java:608)
> >>>> > >> >> >     at org.apache.pig.Main.main(Main.java:156)
> >>>> > >> >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> > >> >> >     at
> >>>> > >> >> >
> >>>> > >> >>
> >>>> > >>
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> > >> >> >     at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> > >> >> >     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> >>>> > >> >> >
> >>>> > >> >> > Is this a problem that caused by hadoop
incompatiblilty?
> Which
> >>>> > >> version of
> >>>> > >> >> > Hadoop is the best?
> >>>> > >> >> >
> >>>> > >> >> > Regards!
> >>>> > >> >> >
> >>>> > >> >> > Chen
> >>>> > >> >> >
> >>>> > >> >> >
> >>>> > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel
Dai <
> >>>> daijy@hortonworks.com
> >>>> > >
> >>>> > >> >> wrote:
> >>>> > >> >> >
> >>>> > >> >> >> Need to set PIG_CLASSPATH:
> >>>> > >> >> >> export
> >>>> PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf
> >>>> > >> >> >>
> >>>> > >> >> >> Thanks,
> >>>> > >> >> >> Daniel
> >>>> > >> >> >>
> >>>> > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen
He <
> airbots@gmail.com>
> >>>> > wrote:
> >>>> > >> >> >> > Any one met this error before?
> >>>> > >> >> >> >
> >>>> > >> >> >> > [main] ERROR org.apache.pig.Main
- ERROR 2998: Unhandled
> >>>> > internal
> >>>> > >> >> error.
> >>>> > >> >> >> > org/apache/tez/common/counters/DAGCounter
> >>>> > >> >> >> >
> >>>> > >> >> >> > Any reply will be appreciated!
> >>>> > >> >> >> >
> >>>> > >> >> >> > Regards!
> >>>> > >> >> >> >
> >>>> > >> >> >> > Chen
> >>>> > >> >> >>
> >>>> > >> >> >> --
> >>>> > >> >> >> CONFIDENTIALITY NOTICE
> >>>> > >> >> >> NOTICE: This message is intended for
the use of the
> >>>> individual or
> >>>> > >> >> entity to
> >>>> > >> >> >> which it is addressed and may contain
information that is
> >>>> > >> confidential,
> >>>> > >> >> >> privileged and exempt from disclosure
under applicable law.
> >>>> If the
> >>>> > >> >> reader
> >>>> > >> >> >> of this message is not the intended
recipient, you are
> hereby
> >>>> > >> notified
> >>>> > >> >> that
> >>>> > >> >> >> any printing, copying, dissemination,
distribution,
> >>>> disclosure or
> >>>> > >> >> >> forwarding of this communication is
strictly prohibited. If
> >>>> you
> >>>> > have
> >>>> > >> >> >> received this communication in error,
please contact the
> >>>> sender
> >>>> > >> >> immediately
> >>>> > >> >> >> and delete it from your system. Thank
You.
> >>>> > >> >> >>
> >>>> > >> >>
> >>>> > >> >> --
> >>>> > >> >> CONFIDENTIALITY NOTICE
> >>>> > >> >> NOTICE: This message is intended for the use
of the
> individual or
> >>>> > >> entity to
> >>>> > >> >> which it is addressed and may contain information
that is
> >>>> > confidential,
> >>>> > >> >> privileged and exempt from disclosure under applicable
law. If
> >>>> the
> >>>> > >> reader
> >>>> > >> >> of this message is not the intended recipient,
you are hereby
> >>>> > notified
> >>>> > >> that
> >>>> > >> >> any printing, copying, dissemination, distribution,
> disclosure or
> >>>> > >> >> forwarding of this communication is strictly
prohibited. If
> you
> >>>> have
> >>>> > >> >> received this communication in error, please
contact the
> sender
> >>>> > >> immediately
> >>>> > >> >> and delete it from your system. Thank You.
> >>>> > >> >>
> >>>> > >>
> >>>> > >> --
> >>>> > >> CONFIDENTIALITY NOTICE
> >>>> > >> NOTICE: This message is intended for the use of the individual
or
> >>>> > entity to
> >>>> > >> which it is addressed and may contain information that
is
> >>>> confidential,
> >>>> > >> privileged and exempt from disclosure under applicable
law. If
> the
> >>>> > reader
> >>>> > >> of this message is not the intended recipient, you are
hereby
> >>>> notified
> >>>> > that
> >>>> > >> any printing, copying, dissemination, distribution, disclosure
or
> >>>> > >> forwarding of this communication is strictly prohibited.
If you
> have
> >>>> > >> received this communication in error, please contact the
sender
> >>>> > immediately
> >>>> > >> and delete it from your system. Thank You.
> >>>> > >>
> >>>> >
> >>>> > --
> >>>> > CONFIDENTIALITY NOTICE
> >>>> > NOTICE: This message is intended for the use of the individual
or
> >>>> entity to
> >>>> > which it is addressed and may contain information that is
> confidential,
> >>>> > privileged and exempt from disclosure under applicable law. If
the
> >>>> reader
> >>>> > of this message is not the intended recipient, you are hereby
> notified
> >>>> that
> >>>> > any printing, copying, dissemination, distribution, disclosure
or
> >>>> > forwarding of this communication is strictly prohibited. If you
have
> >>>> > received this communication in error, please contact the sender
> >>>> immediately
> >>>> > and delete it from your system. Thank You.
> >>>> >
> >>>>
> >>>
> >>>
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message