hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Szita (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-7094) LocalDistributedCacheManager leaves classloaders open, which leaks FDs
Date Tue, 08 May 2018 15:32:00 GMT
Adam Szita created MAPREDUCE-7094:

             Summary: LocalDistributedCacheManager leaves classloaders open, which leaks FDs
                 Key: MAPREDUCE-7094
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Adam Szita

When a user starts a local mapred task from Hive's beeline, it will leave open file descriptors
on the HS2 process (which runs the mapred task).

I debugged this and saw that it is caused by LocalDistributedCacheManager class, which creates
a new URLClassLoader, with a classpath for the two jars seen below. Somewhere down the line
Loaders will be created in this URLClassLoader for these files effectively creating the FD's
on the OS level.

This is never cleaned up after execution, although LocalDistributedCacheManager removes the
files, it will not close the ClassLoader, so FDs are left open although they point to deleted
files at that time:
[root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
java    14439 hive  DEL       REG                8,1             3348748
java    14439 hive  DEL       REG                8,1             3348750
java    14439 hive  649r      REG                8,1   8112438   3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
java    14439 hive  650r      REG                8,1   8112438   3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar


This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

View raw message