hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alice Fan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function
Date Sat, 22 Sep 2018 00:57:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624389#comment-16624389
] 

Alice Fan commented on HIVE-12812:
----------------------------------

Hi [~ctang.ma], Thank you very much for your response :) If you don't mind, I would like to
assign the ticket to myself and complete the patch. I am thinking maybe I can add an if condition
here, at MapReduceCompiler.init(), we only set mapred.input.dir.recursive=true when hive.optimize.union.remove=true.
By this way, it will be focused on addressing the issue caused by the remove union optimizer
and may help to avoid regression in test case.

> Enable mapred.input.dir.recursive by default to support union with aggregate function
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-12812
>                 URL: https://issues.apache.org/jira/browse/HIVE-12812
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1, 2.1.0
>            Reporter: Chaoyu Tang
>            Priority: Major
>         Attachments: HIVE-12812.patch, HIVE-12812.patch, HIVE-12812.patch
>
>
> When union remove optimization is enabled, union query with aggregate function writes
its subquery intermediate results to subdirs which needs mapred.input.dir.recursive to be
enabled in order to be fetched. This property is not defined by default in Hive and often
ignored by user, which causes the query failure and is hard to be debugged.
> So we need set mapred.input.dir.recursive to true whenever union remove optimization
is enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message