hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <>
Subject [jira] [Commented] (HIVE-11878) ClassNotFoundException can possibly occur if multiple jars are registered one at a time in Hive
Date Fri, 27 Nov 2015 16:33:10 GMT


Hive QA commented on HIVE-11878:

Here are the results of testing the latest attachment:

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/' failed with exit status 1 and
output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6145/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7984738 HIVE-12465: Hive might produce wrong results when (outer) joins are
merged (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 7984738 HIVE-12465: Hive might produce wrong results when (outer) joins are
merged (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/
+ /data/hive-ptest/working/scratch/ /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1

This message is automatically generated.

ATTACHMENT ID: 12774196 - PreCommit-HIVE-TRUNK-Build

> ClassNotFoundException can possibly  occur if multiple jars are registered one at a time
in Hive
> ------------------------------------------------------------------------------------------------
>                 Key: HIVE-11878
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1
>            Reporter: Ratandeep Ratti
>            Assignee: Ratandeep Ratti
>              Labels: URLClassLoader
>         Attachments: HIVE-11878 ClassLoader Issues when Registering Jars.pptx, HIVE-11878.2.patch,
HIVE-11878.patch, HIVE-11878_approach3.patch, HIVE-11878_approach3_per_session_clasloader.patch,
HIVE-11878_approach3_with_review_comments.patch, HIVE-11878_approach3_with_review_comments1.patch,
> When we register a jar on the Hive console. Hive creates a fresh URL classloader which
includes the path of the current jar to be registered and all the jar paths of the parent
classloader. The parent classlaoder is the current ThreadContextClassLoader. Once the URLClassloader
is created Hive sets that as the current ThreadContextClassloader.
> So if we register multiple jars in Hive, there will be multiple URLClassLoaders created,
each classloader including the jars from its parent and the one extra jar to be registered.
The last URLClassLoader created will end up as the current ThreadContextClassLoader. (See
details: org.apache.hadoop.hive.ql.exec.Utilities#addToClassPath)
> Now here's an example in which the above strategy can lead to a CNF exception.
> We register 2 jars *j1* and *j2* in Hive console. *j1* contains the UDF class *c1* and
internally relies on class *c2* in jar *j2*. We register *j1* first, the URLClassLoader *u1*
is created and also set as the ThreadContextClassLoader. We register *j2* next, the new URLClassLoader
created will be *u2* with *u1* as parent and *u2* becomes the new ThreadContextClassLoader.
Note *u2* includes paths to both jars *j1* and *j2* whereas *u1* only has paths to *j1* (For
details see: org.apache.hadoop.hive.ql.exec.Utilities#addToClassPath).
> Now when we register class *c1* under a temporary function in Hive, we load the class
using {code} class.forName("c1", true, Thread.currentThread().getContextClassLoader()) {code}
. The currentThreadContext class-loader is *u2*, and it has the path to the class *c1*, but
note that Class-loaders work by delegating to parent class-loader first. In this case class
*c1* will be found and *defined* by class-loader *u1*.
> Now *c1* from jar *j1* has *u1* as its class-loader. If a method (say initialize) is
called in *c1*, which references the class *c2*, *c2* will not be found since the class-loader
used to search for *c2* will be *u1* (Since the caller's class-loader is used to load a class)
> I've added a qtest to explain the problem. Please see the attached patch

This message was sent by Atlassian JIRA

View raw message