tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen He <airb...@gmail.com>
Subject Re: Question about how TezConfiguration
Date Thu, 20 Mar 2014 14:53:10 GMT
Hi Sid

I created Tez-963 which is related to the problem that I met.

Regards!

Chen


On Wed, Mar 19, 2014 at 8:20 PM, Chen He <airbots@gmail.com> wrote:

> Hi Sid
>
> Thank you for the reply. Actually, I think it related to Tez-710 and
> Tez-717. I created TezLocalRunner which extends TezClient. The
> TezLocalRunner need to start DAGAppMaster which is in charge of maintaining
> Tez DAG and let LocalContainerLauncher run tasks in a single
> node.(Tez-710).
>
> Here is the error when LocalContainerLauncher launches task:
>
> java.io.FileNotFoundException: tez-conf.pb (No such file or directory)
>     at java.io.FileInputStream.open(Native Method)
>     at java.io.FileInputStream.<init>(FileInputStream.java:146)
>     at java.io.FileInputStream.<init>(FileInputStream.java:101)
>     at
> org.apache.tez.common.TezUtils.addUserSpecifiedTezConfiguration(TezUtils.java:58)
>     at
> org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:296)
>     at
> org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:263)
>     at java.lang.Thread.run(Thread.java:744)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:744)
>
> I did some research and found out that the tez-conf.pb was successfully
> written to the Tez staging directory. But the YarnTezDagChild can not get
> the value of "TezConfiguration.TEZ_PB_BINARY_CONF_NAME" (get "null")
>
> Any suggestion? Thank you in advance!
>
> Regards!
>
> Chen
>
>
> On Wed, Mar 19, 2014 at 7:50 PM, Siddharth Seth <sseth@apache.org> wrote:
>
>> Right, the configuration provided by the user gets written out and is made
>> available to the AM and tasks.
>>
>> I¹m not sure what you mean by OEI always get ³null²¹ - assuming this is
>> while trying to read a specific key. The Tez client filters out all
>> non-tez configuration parameters before writing out this Configuration -
>> which would cause an empty lookup.
>>
>> Could you describe why you need to pass in a Configuration - typically,
>> I/P/O should have their own payload - which is what they use to configure
>> themselves.
>>
>>
>> On 3/19/14, 3:57 PM, "Chen He" <airbots@gmail.com> wrote:
>>
>> >Dear Tez developers:
>> >
>> >I met a problem when I was working on TezLocalRunner. I need to know how
>> >user's configuration is passed to the YarnTezDagChild so that it can load
>> >it using:
>> >TezUtils.addUserSpecifiedTezConfiguration(conf);
>> >
>> >Here is my understanding:
>> >
>> >1) Users create their TezConfiguration and the TezClientUtils will write
>> >it
>> >to staging directory;
>> >2) DAGAppMaster's main method load TezConfiguration using
>> >TezUtils.addUserSpecifiedTezConfiguration(conf);
>> >3) when the YarnTezDagChild starts to run, its main method also call
>> >TezUtils.addUserSpecifiedTezConfiguration(conf);
>> >
>> >I met a problem that I always get "null" whenever I call the
>> >TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.
>> >
>> >Any reply will be appreciated!
>> >
>> >Regards!
>> >
>> >Chen
>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message