hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21072) NPE when running partitioned CTAS statements
Date Thu, 27 Dec 2018 11:13:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729532#comment-16729532
] 

ASF GitHub Bot commented on HIVE-21072:
---------------------------------------

GitHub user viirya opened a pull request:

    https://github.com/apache/hive/pull/510

    HIVE-21072: Fix NPE when running partitioned CTAS statements

    HIVE-20241 adds support of partitioned CTAS statements:
    
    ```sql
    CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
    SELECT value, key FROM src where key > 200 and key < 300;{code}
     ```
    
    However, I've tried this feature by checking out latest branch-3, and encountered NPE:
    
    ```
    hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
    FAILED: NullPointerException null
    ```
    
    For the added `partition_ctas.q`, it is ran when using `TestMiniLlapLocalCliDriver`. But
the problematic code path is not ran if using `TestMiniLlapLocalCliDriver`. When I go to test
it with TestCliDriver manually, it also throws NullPointerException:
    {code}
    2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main] ql.Driver: FAILED:
NullPointerException null
    java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
        at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
        at org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
        at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
        at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
        at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
        at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
        at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
        at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
        at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
    ```
    
    This patch fixes the NPE issue.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/viirya/hive HIVE-21072

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/hive/pull/510.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #510
    
----
commit 20c397d2a54179bc42fd604f93e6559f0a80e4e6
Author: Liang-Chi Hsieh <viirya@...>
Date:   2018-12-27T10:28:17Z

    Fix NPE when running partitioned CTAS statements.

----


> NPE when running partitioned CTAS statements
> --------------------------------------------
>
>                 Key: HIVE-21072
>                 URL: https://issues.apache.org/jira/browse/HIVE-21072
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Liang-Chi Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> HIVE-20241 adds support of partitioned CTAS statements:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;{code}
>  
> However, I've tried this feature by checking out latest branch-3, and encountered NPE:
> {code:java}
> hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
> FAILED: NullPointerException null
> {code}
> I also ran the query test partition_ctas.q. The test passes when using TestMiniLlapLocalCliDriver,
but when I go to test it with TestCliDriver manually, it also throws NullPointerException:
> {code}
> 2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main] ql.Driver:
FAILED: NullPointerException null
> java.lang.NullPointerException
>     at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
>     at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
>     at org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
>     at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
>     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
>     at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
>     at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
>     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
>     at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
>     at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
>     at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
>     at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
>     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
>     at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
>     at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
>     at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
>     at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
>     at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
>     at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
>     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message