tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bikas Saha <bi...@hortonworks.com>
Subject RE: Question about how TezConfiguration
Date Thu, 20 Mar 2014 16:26:03 GMT
Can we see if we can avoid creating TezLocalRunner that extends TezClient?
We already have TezSession and TezClient and there is an open jira to
unify them. Having a third client side launcher is probably not aligned
with making things easier for the users.


-----Original Message-----
From: Chen He [mailto:airbots@gmail.com]
Sent: Wednesday, March 19, 2014 6:20 PM
To: dev@tez.incubator.apache.org
Subject: Re: Question about how TezConfiguration

Hi Sid

Thank you for the reply. Actually, I think it related to Tez-710 and
Tez-717. I created TezLocalRunner which extends TezClient. The
TezLocalRunner need to start DAGAppMaster which is in charge of
maintaining Tez DAG and let LocalContainerLauncher run tasks in a single

Here is the error when LocalContainerLauncher launches task:

java.io.FileNotFoundException: tez-conf.pb (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileInputStream.<init>(FileInputStream.java:101)
    at java.lang.Thread.run(Thread.java:744)
    at java.lang.Thread.run(Thread.java:744)

I did some research and found out that the tez-conf.pb was successfully
written to the Tez staging directory. But the YarnTezDagChild can not get
the value of "TezConfiguration.TEZ_PB_BINARY_CONF_NAME" (get "null")

Any suggestion? Thank you in advance!



On Wed, Mar 19, 2014 at 7:50 PM, Siddharth Seth <sseth@apache.org> wrote:

> Right, the configuration provided by the user gets written out and is
> made available to the AM and tasks.
> I雋 not sure what you mean by OEI always get 軟ull硃 - assuming this
> is while trying to read a specific key. The Tez client filters out all
> non-tez configuration parameters before writing out this Configuration
> - which would cause an empty lookup.
> Could you describe why you need to pass in a Configuration -
> typically, I/P/O should have their own payload - which is what they
> use to configure themselves.
> On 3/19/14, 3:57 PM, "Chen He" <airbots@gmail.com> wrote:
> >Dear Tez developers:
> >
> >I met a problem when I was working on TezLocalRunner. I need to know
> >how user's configuration is passed to the YarnTezDagChild so that it
> >can load it using:
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >Here is my understanding:
> >
> >1) Users create their TezConfiguration and the TezClientUtils will
> >write it to staging directory;
> >2) DAGAppMaster's main method load TezConfiguration using
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >3) when the YarnTezDagChild starts to run, its main method also call
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >I met a problem that I always get "null" whenever I call the
> >TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.
> >
> >Any reply will be appreciated!
> >
> >Regards!
> >
> >Chen

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message