hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21771) Support partition filter (where clause) in REPL dump command (Bootstrap Dump)
Date Thu, 18 Jul 2019 09:22:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887784#comment-16887784
] 

Hive QA commented on HIVE-21771:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 43s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 54s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 18s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 11s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 28s{color} | {color:blue}
standalone-metastore/metastore-common in master has 31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  3s{color} | {color:blue}
ql in master has 2250 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 36s{color} | {color:blue}
itests/hive-unit in master has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 34s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 26s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 44s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 22s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 22s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 44s{color} | {color:red}
ql: The patch generated 10 new + 570 unchanged - 0 fixed = 580 total (was 570) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 12s{color} | {color:red}
ql generated 9 new + 2241 unchanged - 9 fixed = 2250 total (was 2250) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 42s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 13s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 10s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA239 be a _static_ inner class?
 At HiveParser.java:inner class?  At HiveParser.java:[lines 48849-48862] |
|  |  Dead store to LA31_128 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48589] |
|  |  Dead store to LA31_130 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48602] |
|  |  Dead store to LA31_132 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48615] |
|  |  Dead store to LA31_134 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48628] |
|  |  Dead store to LA31_136 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48641] |
|  |  Dead store to LA31_138 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48654] |
|  |  Dead store to LA31_140 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48667] |
|  |  Dead store to LA31_142 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int,
IntStream)  At HiveParser.java:[line 48680] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18081/dev-support/hive-personality.sh
|
| git revision | master / 8412b37 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18081/yetus/diff-checkstyle-ql.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-18081/yetus/new-findbugs-ql.html
|
| modules | C: standalone-metastore/metastore-common ql itests/hive-unit U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18081/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Support partition filter (where clause) in REPL dump command (Bootstrap Dump)
> -----------------------------------------------------------------------------
>
>                 Key: HIVE-21771
>                 URL: https://issues.apache.org/jira/browse/HIVE-21771
>             Project: Hive
>          Issue Type: Sub-task
>          Components: HiveServer2, repl
>    Affects Versions: 4.0.0
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>         Attachments: HIVE-21771.01.patch, HIVE-21771.02.patch
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> *Bootstrap for managed table*
> User should be allowed to execute REPL DUMP with where clause. The where clause should
support filtering out partition from dump. Format of the where clause should be similar to
*"REPL DUMP dbname from 10 where 't0' where key < 10,'t1'* where key = 3, '(t2*)|'t3' where
key > 3".* For initial version, very basic filter condition will be supported and later
the complexity will be increased as and when required.
>  * From the AST generated for the where clause, extract the table information.
>  * Generate AST for each table.
>  * List the partition for each table using the AST generated for each table using the
  same metastore API used by select query.
>  * During bootstrap load use the partition list to dump the partitions.
>  * During incremental dump, use the list to filter out the event.
> In case of bootstrap load, all the tables of the database will be scanned and
>  * If table is not partitioned, then it will be dumped.
>  * If key provided in the filter condition for the table is not a partition column, then
dump will fail.
>  * If table is not mentioned in the where clause, then all partitions of the table will
be dumped.
>  * All the partitioned of the table satisfying the where clause will be dumped.
> *Incremental for managed table (Not part of this patch)*
> In case of Incremental Dump, the events from the notification log will be scanned and
once the partition spec is extracted from the event, the partition spec will be filtered
against the condition.
>  * If table is not partitioned then the event will be added to the dump.
>  * If key mentioned is not a partition column, then dump will fail.
>  * If the table is not mentioned in the filter then event will be added to the dump.
>  * If the event is multi partitioned, then the event will be added to the dump. (Filtering
out redundant partitions from message will be done as part of separate task).
>  * If the partition spec matches the filter, then the event will be added to the dump*.*
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message