sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Lin via Review Board <nore...@reviews.apache.org>
Subject Re: Review Request 54528: SQOOP-3042 - Sqoop does not clear compile directory under /tmp/sqoop-<username>/compile automatically
Date Thu, 12 Jul 2018 08:04:48 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated July 12, 2018, 8:04 a.m.)

Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Updated docs and added test case.

Thanks for your suggestions.

Bugs: SQOOP-3042

Repository: sqoop-trunk


After running sqoop, all the temp files generated by ClassWriter are left behind on disk,
so anyone can check those JAVA files to see the schema of those tables that Sqoop has been
interacting with. By default, the directory is under /tmp/sqoop-<username>/compile.

In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see that we did add
"deleteOnExit" on the temp dir:
    for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
      hashDir = new File(baseDir, RandomHash.generateMD5String());
      while (hashDir.exists()) {
        hashDir = new File(baseDir, RandomHash.generateMD5String());

      if (hashDir.mkdirs()) {
        // We created the directory. Use it.
        // If this directory is not actually filled with files, delete it
        // when the JVM quits.
However, I believe it failed to delete due to directory is not empty.

Diffs (updated)

  src/docs/man/codegen-args.txt c1afaef9 
  src/docs/user/common-args.txt 98f19be0 
  src/java/org/apache/sqoop/SqoopOptions.java 3a19aeac 
  src/java/org/apache/sqoop/orm/ClassWriter.java a4a768af 
  src/java/org/apache/sqoop/orm/CompilationManager.java 6590cacc 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java 8d318327 
  src/java/org/apache/sqoop/util/DirCleanupHook.java PRE-CREATION 
  src/test/org/apache/sqoop/util/TestDirCleanupHook.java PRE-CREATION 

Diff: https://reviews.apache.org/r/54528/diff/6/

Changes: https://reviews.apache.org/r/54528/diff/5-6/


I have tested manually. I have checked with a couple of other Java developers and it turned
out that it is not easy to add test for deleteOnExit, so I did not add any test cases. The
code path I changed does not seem to have test coverage either. Let me know if I am wrong.



Eric Lin

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message