hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14688) Hive drop call fails in presence of TDE
Date Fri, 16 Dec 2016 03:33:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753313#comment-15753313
] 

Hive QA commented on HIVE-14688:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12843504/HIVE-14688.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10803 tests executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=234)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=123)
	[groupby_complex_types.q,multigroupby_singlemr.q,mapjoin_decimal.q,groupby7.q,join5.q,bucketmapjoin_negative2.q,vectorization_div0.q,union_script.q,add_part_multiple.q,limit_pushdown.q,union_remove_17.q,uniquejoin.q,metadata_only_queries_with_filters.q,union25.q,load_dyn_part13.q]
TestVectorizedColumnReaderBase - did not produce a TEST-*.xml file (likely timed out) (batchId=251)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_sort_array] (batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (batchId=133)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1] (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=151)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2602/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2602/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2602/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12843504 - PreCommit-HIVE-Build

> Hive drop call fails in presence of TDE
> ---------------------------------------
>
>                 Key: HIVE-14688
>                 URL: https://issues.apache.org/jira/browse/HIVE-14688
>             Project: Hive
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 1.2.1, 2.0.0
>            Reporter: Deepesh Khandelwal
>            Assignee: Wei Zheng
>         Attachments: HIVE-14688.1.patch
>
>
> In Hadoop 2.8.0 TDE trash collection was fixed through HDFS-8831. This enables us to
make drop table calls for Hive managed tables where Hive metastore warehouse directory is
in encrypted zone. However even with the feature in HDFS, Hive drop table currently fail:
> {noformat}
> $ hdfs crypto -listZones
> /apps/hive/warehouse  key2 
> $ hdfs dfs -ls /apps/hive/warehouse
> Found 1 items
> drwxrwxrwt   - hdfs hdfs          0 2016-09-01 02:54 /apps/hive/warehouse/.Trash
> hive> create table abc(a string, b int);
> OK
> Time taken: 5.538 seconds
> hive> dfs -ls /apps/hive/warehouse;
> Found 2 items
> drwxrwxrwt   - hdfs   hdfs          0 2016-09-01 02:54 /apps/hive/warehouse/.Trash
> drwxrwxrwx   - deepesh hdfs          0 2016-09-01 17:15 /apps/hive/warehouse/abc
> hive> drop table if exists abc;
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable
to drop default.abc because it is in an encryption zone and trash is enabled.  Use PURGE option
to skip trash.)
> {noformat}
> The problem lies here:
> {code:title=metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java}
> private void checkTrashPurgeCombination(Path pathToData, String objectName, boolean ifPurge)
> ...
>       if (trashEnabled) {
>         try {
>           HadoopShims.HdfsEncryptionShim shim =
>             ShimLoader.getHadoopShims().createHdfsEncryptionShim(FileSystem.get(hiveConf),
hiveConf);
>           if (shim.isPathEncrypted(pathToData)) {
>             throw new MetaException("Unable to drop " + objectName + " because it is
in an encryption zone" +
>               " and trash is enabled.  Use PURGE option to skip trash.");
>           }
>         } catch (IOException ex) {
>           MetaException e = new MetaException(ex.getMessage());
>           e.initCause(ex);
>           throw e;
>         }
>       }
> {code}
> As we can see that we are making an assumption that delete wouldn't be successful in
encrypted zone. We need to modify this logic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message