toree-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lrese...@apache.org
Subject incubator-toree git commit: [MINOR] Adds file tagging for z/os platforms
Date Thu, 07 Sep 2017 15:04:31 GMT
Repository: incubator-toree
Updated Branches:
  refs/heads/master 9cdc29860 -> db30f7292


[MINOR] Adds file tagging for z/os platforms

On a z/OS system, when copying a python file to tmp,
python is unable to execute scripts unless it is tagged
with the appropriate file encoding.

Closes #136


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/db30f729
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/db30f729
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/db30f729

Branch: refs/heads/master
Commit: db30f729245c357d8014d5bc21c94d71770b18ae
Parents: 9cdc298
Author: Yunli Tang <tangy@us.ibm.com>
Authored: Tue Aug 29 12:40:53 2017 -0400
Committer: Luciano Resende <lresende@apache.org>
Committed: Thu Sep 7 07:59:32 2017 -0700

----------------------------------------------------------------------
 .../kernel/interpreter/pyspark/PySparkProcess.scala   | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/db30f729/pyspark-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala
----------------------------------------------------------------------
diff --git a/pyspark-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala
b/pyspark-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala
index d89865a..a97a243 100644
--- a/pyspark-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala
+++ b/pyspark-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala
@@ -24,6 +24,7 @@ import org.apache.commons.exec._
 import org.apache.commons.io.IOUtils
 import org.apache.spark.SparkContext
 import org.slf4j.LoggerFactory
+import sys.process._
 
 /**
  * Represents the Python process used to evaluate PySpark code.
@@ -89,4 +90,17 @@ class PySparkProcess(
       "PYTHONPATH" -> updatedPythonPath
     )
   }
+
+  override protected def copyResourceToTmp(resource: String): String = {
+    val destination = super.copyResourceToTmp(resource)
+    if (System.getProperty("os.name").equals("z/OS")){
+        tagPySparkResource(destination)
+    }
+    destination 
+  }
+
+  private def tagPySparkResource(destPath: String): Unit = {
+      val exitCode = Seq("chtag", "-t", "-c", "ISO8859-1", destPath).!
+      if (exitCode != 0) logger.warn("PySpark resource was not tagged correctly.")
+  }
 }


Mime
View raw message