hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wang Haihua (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function
Date Thu, 14 Jun 2018 17:31:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512794#comment-16512794
] 

Wang Haihua commented on HIVE-12812:
------------------------------------

[[~kgyrtkirk]]  We use hive 1.2.1, just in fact, mapreduce is though old but always be the
most stable engine for Hive. We use hive heavily in production( expand to 5K nodes+), which
consist 60% tez, 35% mapreduce(for corner case tez cannot handled), and 5% SparkSQL(the future). 

So the first priority is stable(include data quality and consistency, SLA and so on), we put
much patch on our version of Hive.

As tez and mapreduce is mixed, we encountered this issue. And i suppose this is a common
issue on production , for reason that HiveOnSpark, HiveOnTez is not stable enough.

 

> Enable mapred.input.dir.recursive by default to support union with aggregate function
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-12812
>                 URL: https://issues.apache.org/jira/browse/HIVE-12812
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1, 2.1.0
>            Reporter: Chaoyu Tang
>            Priority: Major
>         Attachments: HIVE-12812.patch, HIVE-12812.patch, HIVE-12812.patch
>
>
> When union remove optimization is enabled, union query with aggregate function writes
its subquery intermediate results to subdirs which needs mapred.input.dir.recursive to be
enabled in order to be fetched. This property is not defined by default in Hive and often
ignored by user, which causes the query failure and is hard to be debugged.
> So we need set mapred.input.dir.recursive to true whenever union remove optimization
is enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message