sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-<username>/compile automatically
Date Fri, 18 Jan 2019 11:40:00 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746202#comment-16746202
] 

Vani commented on SQOOP-3042:
-----------------------------

[~vasas] [~dionusos]

"The error means that the generated source file cannot be moved to the code output directory,
which is the current directory by default but can be overridden with the - outdir option".
But this doesn't take care of the below scenarios:-

1) Technically if the "--outdir" is not specified in the sqoop command, this would always
result in error message logged in sqoop log if the current directory has already a file with
the same name.
2) Since this is not really an error, rather a product behavior shouldn’t it be logged as
Info message?
3) As better product design, new behavior should not change user experience. Following impacts
are observed:
A) Error message log which is a product design.
B) To mitigate the error message or having this actually worked, now user has to modify all
the commands manually.

Severity levels are used by many user in their monitoring system and such changes should be
thoroughly rationalized.

Please let me know your thoughts

> Sqoop does not clear compile directory under /tmp/sqoop-<username>/compile automatically
> ----------------------------------------------------------------------------------------
>
>                 Key: SQOOP-3042
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3042
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.6
>            Reporter: Eric Lin
>            Assignee: Eric Lin
>            Priority: Critical
>              Labels: patch
>             Fix For: 3.0.0
>
>         Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, SQOOP-3042.4.patch, SQOOP-3042.5.patch,
SQOOP-3042.6.patch, SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left behind on disk,
so anyone can check those JAVA files to see the schema of those tables that Sqoop has been
interacting with. By default, the directory is under /tmp/sqoop-<username>/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see that we
did add "deleteOnExit" on the temp dir:
> {code}
>     for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>       hashDir = new File(baseDir, RandomHash.generateMD5String());
>       while (hashDir.exists()) {
>         hashDir = new File(baseDir, RandomHash.generateMD5String());
>       }
>       if (hashDir.mkdirs()) {
>         // We created the directory. Use it.
>         // If this directory is not actually filled with files, delete it
>         // when the JVM quits.
>         hashDir.deleteOnExit();
>         break;
>       }
>     }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message