hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aegeaner (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HIVE-18177) Error creating temporary staging folder on HDFS when creating Hive table
Date Fri, 01 Dec 2017 16:42:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-18177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aegeaner resolved HIVE-18177.
-----------------------------
      Resolution: Won't Fix
    Release Note: Configuration problem. The aux lib dir under $HIVE_HOME lacks packages.

> Error creating temporary staging folder on HDFS when creating Hive table
> ------------------------------------------------------------------------
>
>                 Key: HIVE-18177
>                 URL: https://issues.apache.org/jira/browse/HIVE-18177
>             Project: Hive
>          Issue Type: Bug
>          Components: Parser
>    Affects Versions: 2.3.1
>            Reporter: Aegeaner
>
> When creating a table in hive using statement:
> {code:java}
> create database if not exists ${DB};
> use ${DB};
> drop table if exists date_dim;
> create table date_dim
> stored as ${FILE}
> as select * from ${SOURCE}.date_dim;
> {code}
> The statement execution failed as:
> {code:java}
> FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db.
Error encountered near token 'TOK_TMP_FILE'
> FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db.
Error encountered near token 'TOK_TMP_FILE'
> {code}
> We got this exception stackļ¼š
> {code:java}
> 2017-11-29T17:32:47,646  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner:
Completed phase 1 of Semantic Analysis
> 2017-11-29T17:32:47,646  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner:
Get metadata for source tables
> 2017-11-29T17:32:47,646  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] metastore.HiveMetaStore:
0: get_table : db=tpcds_text_2 tbl=date_dim
> 2017-11-29T17:32:47,647  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] HiveMetaStore.audit:
ugi=hadoop	ip=unknown-ip-addr	cmd=get_table : db=tpcds_text_2 tbl=date_dim
> 2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner:
Get metadata for subqueries
> 2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner:
Get metadata for destination tables
> 2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] metastore.HiveMetaStore:
0: get_database: tpcds_bin_partitioned_orc_2
> 2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] HiveMetaStore.audit:
ugi=hadoop	ip=unknown-ip-addr	cmd=get_database: tpcds_bin_partitioned_orc_2
> 2017-11-29T17:32:48,308  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] common.FileUtils:
Creating directory if it doesn't exist: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db/.hive-staging_hive_2017-11-29_17-32-47_541_2322222506518783479-1
> 2017-11-29T17:32:48,330 ERROR [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner:
org.apache.hadoop.hive.ql.parse.SemanticException: 0:0 Error creating temporary folder on:
hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db. Error encountered near
token 'TOK_TMP_FILE'
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2211)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11080)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11133)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> 	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
> 	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
> 	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: java.lang.RuntimeException: Cannot create staging directory 'hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db/.hive-staging_hive_2017-11-29_17-32-47_541_2322222506518783479-1':
Invalid host name: local host is: (unknown); destination host is: "ns-offline":8020; java.net.UnknownHostException;
For more details see:  http://wiki.apache.org/hadoop/UnknownHost
> 	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:374)
> 	at org.apache.hadoop.hive.ql.Context.getExtTmpPathRelTo(Context.java:632)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2208)
> 	... 25 more
> Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown);
destination host is: "ns-offline":8020; java.net.UnknownHostException; For more details see:
 http://wiki.apache.org/hadoop/UnknownHost
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
> 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:744)
> 	at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:445)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1373)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1337)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> 	at com.sun.proxy.$Proxy36.getFileInfo(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:787)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
> 	at com.sun.proxy.$Proxy37.getFileInfo(Unknown Source)
> 	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1700)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1436)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1433)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1433)
> 	at org.apache.hadoop.hive.common.FileUtils.mkdir(FileUtils.java:528)
> 	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:366)
> 	... 27 more
> Caused by: java.net.UnknownHostException
> 	... 52 more
> 2017-11-29T17:32:48,331 ERROR [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] ql.Driver:
FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db.
Error encountered near token 'TOK_TMP_FILE'
> org.apache.hadoop.hive.ql.parse.SemanticException: 0:0 Error creating temporary folder
on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db. Error encountered
near token 'TOK_TMP_FILE'
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2211)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11080)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11133)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> 	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
> 	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
> 	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: java.lang.RuntimeException: Cannot create staging directory 'hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db/.hive-staging_hive_2017-11-29_17-32-47_541_2322222506518783479-1':
Invalid host name: local host is: (unknown); destination host is: "ns-offline":8020; java.net.UnknownHostException;
For more details see:  http://wiki.apache.org/hadoop/UnknownHost
> 	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:374)
> 	at org.apache.hadoop.hive.ql.Context.getExtTmpPathRelTo(Context.java:632)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2208)
> 	... 25 more
> Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown);
destination host is: "ns-offline":8020; java.net.UnknownHostException; For more details see:
 http://wiki.apache.org/hadoop/UnknownHost
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
> 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:744)
> 	at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:445)
> 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1373)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1337)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> 	at com.sun.proxy.$Proxy36.getFileInfo(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:787)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
> 	at com.sun.proxy.$Proxy37.getFileInfo(Unknown Source)
> 	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1700)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1436)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1433)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1433)
> 	at org.apache.hadoop.hive.common.FileUtils.mkdir(FileUtils.java:528)
> 	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:366)
> 	... 27 more
> Caused by: java.net.UnknownHostException
> 	... 52 more
> {code}
> In HADOOP core-site.xml:
> {code:java}
> <property>
>   <name>fs.default.name</name>
>   <value>hdfs://ns-offline</value>
> </property>
> {code}
> The Exception is thrown at Hive QL:
> [https://github.com/apache/hive/blob/07fe7e210cb444aec43cb5adda37f8f7cd26f243/ql/src/java/org/apache/hadoop/hive/ql/Context.java#L396]
> And called HDFS mkdir operation at:
> [https://github.com/apache/hive/blob/07fe7e210cb444aec43cb5adda37f8f7cd26f243/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L579]
> The call above took a _+conf+_ parameter but not used.
> And also, in PathInfo class where we got the FileSystem instanceļ¼š
> [https://github.com/apache/hive/blob/32e854ef1c25f21d53f7932723cfc76bf75a71cd/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/PathInfo.java#L69]
> {code:java}
> FileSystem fileSystem = inputPath.getFileSystem(hiveConf);
> {code}
> We passed  a hive configuration instead of HDFS config.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message