tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Xi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3728) When using HDFS federation, setting tez.simple.history.logging.dir to viewfs:// causes AM to fail
Date Mon, 15 May 2017 07:47:04 GMT
Chen Xi created TEZ-3728:
----------------------------

             Summary: When using HDFS federation, setting tez.simple.history.logging.dir to
viewfs:// causes AM to fail
                 Key: TEZ-3728
                 URL: https://issues.apache.org/jira/browse/TEZ-3728
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.8.5
         Environment: hive1.1.0
hadoop2.6.0
            Reporter: Chen Xi


tez.simple.history.logging.dir does not support viewfs schema.
In tez-site.xml we have:
{noformat}
<property>
    <name>tez.simple.history.logging.dir</name>
    <value>viewfs://clusterX/history-tez</value>
</property>
{noformat}
TezAM throws such exception:
{noformat}
2017-05-15 15:34:00,809 [INFO] [main] |impl.SimpleHistoryLoggingService|: Using configured
log file location for SimpleHistoryLoggingService logDirPath=viewfs://clusterX/history-tez
2017-05-15 15:34:00,812 [INFO] [main] |impl.SimpleHistoryLoggingService|: Initializing SimpleHistoryLoggingService,
logFileLocation=hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001, maxErrors=10
2017-05-15 15:34:00,812 [INFO] [main] |recovery.RecoveryService|: RecoveryService initialized
with recoveryPath=hdfs://ns2/tmp/hive/op/_tez_session_dir/6b76d2f7-2f00-4f0c-9a15-4ad8478cf32e/.tez/application_1494315174208_1108/recovery/1,
bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2017-05-15 15:34:00,814 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:APP_LAUNCHED]:
applicationId=application_1494315174208_1108, appSubmitTime=1494833615647, launchTime=1494833639649
2017-05-15 15:34:00,815 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:AM_LAUNCHED]:
appAttemptId=appattempt_1494315174208_1108_000001, appSubmitTime=1494833615647, launchTime=1494833639649
2017-05-15 15:34:00,818 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler]
|impl.SimpleHistoryLoggingService|: Starting SimpleHistoryLoggingService
2017-05-15 15:34:00,819 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler]
|service.AbstractService|: Service org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
failed in state STARTED; cause: java.lang.IllegalArgumentException: Wrong FS: hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001,
expected: viewfs://clusterX/
java.lang.IllegalArgumentException: Wrong FS: hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001,
expected: viewfs://clusterX/
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
	at org.apache.hadoop.fs.viewfs.ViewFileSystem.getUriPath(ViewFileSystem.java:117)
	at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:615)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
	at org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService.serviceStart(SimpleHistoryLoggingService.java:96)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
	at org.apache.tez.dag.history.HistoryEventHandler.serviceStart(HistoryEventHandler.java:105)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1838)
	at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1859)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message