hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
Date Fri, 14 Dec 2018 07:11:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721008#comment-16721008
] 

Hive QA commented on HIVE-20992:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951748/HIVE-20992.4.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15317/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15317/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15317/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ date '+%Y-%m-%d %T.%3N'
2018-12-14 07:09:07.167
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15317/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-12-14 07:09:07.171
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e8e0396 HIVE-21020: log which table/partition is being processed by a txn in
Worker (Eugene Koifman, reviewed by Jason Dere, Igor Kryvenko)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at e8e0396 HIVE-21020: log which table/partition is being processed by a txn in
Worker (Eugene Koifman, reviewed by Jason Dere, Igor Kryvenko)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-12-14 07:09:08.283
+ rm -rf ../yetus_PreCommit-HIVE-Build-15317
+ mkdir ../yetus_PreCommit-HIVE-Build-15317
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15317
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15317/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8193302175318580749.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc8193302175318580749.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator  Version 3.5.2
protoc-jar: executing: [/tmp/protoc276867551589549715.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator  Version 3.5.2
Output file /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 41 classes.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-spark-client: Execution process-resource-bundles
of goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed. ConcurrentModificationException
-> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following
articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hive-spark-client
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-15317
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12951748 - PreCommit-HIVE-Build

> Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
> --------------------------------------------------------------------------------------
>
>                 Key: HIVE-20992
>                 URL: https://issues.apache.org/jira/browse/HIVE-20992
>             Project: Hive
>          Issue Type: Improvement
>          Components: Metastore, Security, Standalone Metastore
>    Affects Versions: 4.0.0
>            Reporter: Morio Ramdenbourg
>            Assignee: Morio Ramdenbourg
>            Priority: Minor
>         Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, HIVE-20992.4.patch, HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS Service to the
HMSDB by configuring the following two properties:
>  # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC metastore. To use
SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection
URL. (E.g. "jdbc:postgresql://myhost/db?ssl=true")
>  # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties for metastore
to access database when JDO connection URL. (E.g. javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd)
> However, the latter configuration option is opaque and poses some problems. The most
glaring of which is it takes in _any_ [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html]
system property, whether it is [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization]
or not. This can cause some unintended side-effects for other components of the HMS, especially
if it overrides an already-set system property. If the user truly wishes to add an unrelated
Java property, setting it statically using the "-D" option of the _java_ command is more appropriate.
Secondly, the truststore password is stored in plain text. We should add Hadoop Shims back
to the HMS to prevent exposing these passwords, but this effort can be done after this ticket.
> I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the following
properties:
>  * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_*
>  ** Set this to true to for using SSL/TLS encryption from the HMS Service to the HMS
backend store
>  ** Default: false
>  * _*hive.metastore.dbaccess.ssl.truststore.path (metastore.dbaccess.ssl.truststore.path)*_
>  ** Truststore location
>  ** Directly maps to _javax.net.ssl.trustStore_ System property
>  ** Default: None
>  ** E.g. _/tmp/truststore_
>  * *_hive.metastore.dbaccess.ssl.truststore.password (metastore.dbaccess.ssl.truststore.password)_*
>  ** Truststore password
>  ** Directly maps to _javax.net.ssl.trustStorePassword_ System property
>  ** Default: None
>  ** E.g. _password_
>  * *_hive.metastore.dbaccess.ssl.truststore.type (metastore.dbaccess.ssl.truststore.type)_*
>  ** Truststore type
>  ** Directly maps to _javax.net.ssl.trustStoreType_ System property
>  ** Default: JKS
>  ** E.g. _pkcs12_
> We should guide the user towards an easier TLS configuration experience. This is the
minimum configuration necessary to configure TLS to the HMSDB. If we need other options,
such as the keystore location/password for dual-authentication, then we can add those on afterwards.
> Also, document these changes - [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL]
does not have up-to-date documentation, and these new parameters will need documentation as
well.
> Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message