tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen He <airb...@gmail.com>
Subject Re: Question about how TezConfiguration
Date Thu, 20 Mar 2014 01:20:27 GMT
Hi Sid

Thank you for the reply. Actually, I think it related to Tez-710 and
Tez-717. I created TezLocalRunner which extends TezClient. The
TezLocalRunner need to start DAGAppMaster which is in charge of maintaining
Tez DAG and let LocalContainerLauncher run tasks in a single
node.(Tez-710).

Here is the error when LocalContainerLauncher launches task:

java.io.FileNotFoundException: tez-conf.pb (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileInputStream.<init>(FileInputStream.java:101)
    at
org.apache.tez.common.TezUtils.addUserSpecifiedTezConfiguration(TezUtils.java:58)
    at
org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:296)
    at
org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:263)
    at java.lang.Thread.run(Thread.java:744)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

I did some research and found out that the tez-conf.pb was successfully
written to the Tez staging directory. But the YarnTezDagChild can not get
the value of "TezConfiguration.TEZ_PB_BINARY_CONF_NAME" (get "null")

Any suggestion? Thank you in advance!

Regards!

Chen


On Wed, Mar 19, 2014 at 7:50 PM, Siddharth Seth <sseth@apache.org> wrote:

> Right, the configuration provided by the user gets written out and is made
> available to the AM and tasks.
>
> I¹m not sure what you mean by OEI always get ³null²¹ - assuming this is
> while trying to read a specific key. The Tez client filters out all
> non-tez configuration parameters before writing out this Configuration -
> which would cause an empty lookup.
>
> Could you describe why you need to pass in a Configuration - typically,
> I/P/O should have their own payload - which is what they use to configure
> themselves.
>
>
> On 3/19/14, 3:57 PM, "Chen He" <airbots@gmail.com> wrote:
>
> >Dear Tez developers:
> >
> >I met a problem when I was working on TezLocalRunner. I need to know how
> >user's configuration is passed to the YarnTezDagChild so that it can load
> >it using:
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >Here is my understanding:
> >
> >1) Users create their TezConfiguration and the TezClientUtils will write
> >it
> >to staging directory;
> >2) DAGAppMaster's main method load TezConfiguration using
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >3) when the YarnTezDagChild starts to run, its main method also call
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >I met a problem that I always get "null" whenever I call the
> >TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.
> >
> >Any reply will be appreciated!
> >
> >Regards!
> >
> >Chen
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message