hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20220) Incorrect result when hive.groupby.skewindata is enabled
Date Mon, 30 Jul 2018 06:04:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561455#comment-16561455
] 

Hive QA commented on HIVE-20220:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 39s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 11s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 19s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 51s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 31s{color} | {color:blue}
common in master has 64 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 57s{color} | {color:blue}
ql in master has 2297 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  9s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  9s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 46s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 23s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 23s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 16s{color} | {color:red}
common: The patch generated 2 new + 423 unchanged - 0 fixed = 425 total (was 423) {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 38s{color} | {color:red}
ql: The patch generated 1 new + 55 unchanged - 0 fixed = 56 total (was 55) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 48s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 10s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 13s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 44s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-12938/dev-support/hive-personality.sh
|
| git revision | master / 83e5397 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-12938/yetus/diff-checkstyle-common.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-12938/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-12938/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Incorrect result when hive.groupby.skewindata is enabled
> --------------------------------------------------------
>
>                 Key: HIVE-20220
>                 URL: https://issues.apache.org/jira/browse/HIVE-20220
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 3.0.0
>            Reporter: Ganesha Shreedhara
>            Assignee: Ganesha Shreedhara
>            Priority: Major
>         Attachments: HIVE-20220.patch
>
>
> hive.groupby.skewindata makes use of rand UDF to randomly distribute grouped by keys
to the reducers and hence avoids overloading a single reducer when there is a skew in data. 
> This random distribution of keys is buggy when the reducer fails to fetch the mapper
output due to a faulty datanode or any other reason. When reducer finds that it can't fetch
mapper output, it sends a signal to Application Master to reattempt the corresponding map
task. The reattempted map task will now get the different random value from rand function
and hence the keys that gets distributed now to the reducer will not be same as the previous
run. 
>  
> *Steps to reproduce:*
> create table test(id int);
> insert into test values (1),(2),(2),(3),(3),(3),(4),(4),(4),(4),(5),(5),(5),(5),(5),(6),(6),(6),(6),(6),(6),(7),(7),(7),(7),(7),(7),(7),(7),(8),(8),(8),(8),(8),(8),(8),(8),(9),(9),(9),(9),(9),(9),(9),(9),(9);
> SET hive.groupby.skewindata=true;
> SET mapreduce.reduce.reduces=2;
> //Add a debug port for reducer
> select count(1) from test group by id;
> //Remove mapper's intermediate output file when map stage is completed and one out of
2 reduce tasks is completed and then continue the run. This causes 2nd reducer to send event
to Application Master to rerun the map task. 
> The following is the expected result. 
> 1
> 2
> 3
> 4
> 5
> 6
> 8
> 8
> 9 
>  
> But you may get different result due to a different value returned by the rand function
in the second run causing different distribution of keys.
> This needs to be fixed such that the mapper distributes the same keys even if it is
reattempted multiple times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message