hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
Date Wed, 13 Dec 2017 23:19:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290076#comment-16290076
] 

Jason Dere commented on HIVE-18271:
-----------------------------------

Dug into some of the details with [~ashutoshc] and [~bslim]. The FileSinkOperator should not
even be trying to call Utilities.mvToFinalPath(), because it is a non-native table - there
is actually logic in place for that. The problem is that FileSinkOperator.isNativeTable is
dependent on being set during initializeOp() .. and it appears that initializeOp() is never
being called for the operators in the TezTask (client side)! Making a patch to fix the immediate
problem, by making isNativeTable into a method, that does not depend on initializeOp().

We may want to look into whether the operators within TezTask should have initialize() called
on them as a future item.

> Druid Insert into fails with exception when committing files
> ------------------------------------------------------------
>
>                 Key: HIVE-18271
>                 URL: https://issues.apache.org/jira/browse/HIVE-18271
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Nishant Bangarwa
>            Assignee: Nishant Bangarwa
>             Fix For: 3.0.0
>
>         Attachments: HIVE-18271.patch
>
>
> Exception - 
> {code}
> 000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to: hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move: hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
to: hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> 	at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> 	at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> 	at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> 	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> 	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message