sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Kemper (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-<username>/compile automatically
Date Thu, 20 Jul 2017 13:18:00 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094664#comment-16094664
] 

Markus Kemper commented on SQOOP-3042:
--------------------------------------

As a possible workaround the Sqoop Environment script could be used to manage the contents
of the compile dir this.

Sample Code
{noformat}
cat /etc/sqoop/conf/sqoop-env.sh 
echo
echo SQOOP COMPILE DIR DIRECTORY COUNT
ls -ltd /tmp/sqoop*/compile/* | wc
echo SQOOP COMPILE DIR FILES COUNT
ls -ltR /tmp/sqoop*/compile/* | wc
echo SQOOP COMPILE DIR OLDEST DIRECTORY
ls -ltd /tmp/sqoop*/compile/* | tail -1
echo SQOOP COMPILE DIR TMPWATCH 60d DELETE TEST
tmpwatch --test 60d /tmp/sqoop*/compile/* | wc
echo SQOOP COMPILE DIR TMPWATCH 60d DELETE
tmpwatch 60d /tmp/sqoop*/compile/* | wc
echo SQOOP COMPILE DIR DIRECTORY COUNT
ls -ltd /tmp/sqoop*/compile/* | wc
echo SQOOP COMPILE DIR FILES COUNT
ls -ltR /tmp/sqoop*/compile/* | wc
echo SQOOP COMPILE DIR OLDEST DIRECTORY
ls -ltd /tmp/sqoop*/compile/* | tail -1
echo
{noformat}

*Sample Output*
{noformat}
$ sqoop version
SQOOP COMPILE DIR DIRECTORY COUNT
   1410   12690  142458
SQOOP COMPILE DIR FILES COUNT
   6853   27702  247866
SQOOP COMPILE DIR OLDEST DIRECTORY
drwxrwxr-x 2 user1 user1  4096 Dec  6  2016 /tmp/sqoop-user1/compile/734bf9f8090d24a8757303c44faf2825
SQOOP COMPILE DIR TMPWATCH 60d DELETE TEST
      0       0       0
SQOOP COMPILE DIR TMPWATCH 60d DELETE
      0       0       0
SQOOP COMPILE DIR DIRECTORY COUNT
   1410   12690  142458
SQOOP COMPILE DIR FILES COUNT
   6853   27702  247866
SQOOP COMPILE DIR OLDEST DIRECTORY
drwxrwxr-x 2 user1 user1  4096 Dec  6  2016 /tmp/sqoop-user1/compile/734bf9f8090d24a8757303c44faf2825

Warning: /opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/bin/../lib/sqoop/../accumulo does
not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/07/20 06:16:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.1
Sqoop 1.4.6-cdh5.10.1
git commit id 
Compiled by jenkins on Mon Mar 20 02:34:12 PDT 2017
{noformat}

> Sqoop does not clear compile directory under /tmp/sqoop-<username>/compile automatically
> ----------------------------------------------------------------------------------------
>
>                 Key: SQOOP-3042
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3042
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.6
>            Reporter: Eric Lin
>            Assignee: Eric Lin
>            Priority: Critical
>              Labels: patch
>         Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, SQOOP-3042.4.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left behind on disk,
so anyone can check those JAVA files to see the schema of those tables that Sqoop has been
interacting with. By default, the directory is under /tmp/sqoop-<username>/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see that we
did add "deleteOnExit" on the temp dir:
> {code}
>     for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>       hashDir = new File(baseDir, RandomHash.generateMD5String());
>       while (hashDir.exists()) {
>         hashDir = new File(baseDir, RandomHash.generateMD5String());
>       }
>       if (hashDir.mkdirs()) {
>         // We created the directory. Use it.
>         // If this directory is not actually filled with files, delete it
>         // when the JVM quits.
>         hashDir.deleteOnExit();
>         break;
>       }
>     }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message