spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From innowireless TaeYun Kim <>
Subject Suggestion for SPARK-1825
Date Tue, 22 Jul 2014 01:16:07 GMT


A couple of month ago, I made a pull request to fix

My pull request is here: 


But that pull request has problems:

l  It is Hadoop 2.4.0+ only. It won't compile on the versions below it.

l  The related Hadoop API is marked as '@Unstable'.


Here is an idea to remedy the problems: a new Spark configuration variable.

Maybe it can be named as "spark.yarn.submit.crossplatform".

If it is set to "true"(default is false), the related Spark code can use the
hard-coded strings that is the same as the Hadoop API provides, thus
avoiding compile error on the Hadoop versions below 2.4.0.


Can someone implement this feature, if this idea is acceptable?

Currently my knowledge on Spark source code and Scala is limited to
implement it myself.

To the right person, the modification should be trivial.

You can refer to the source code changes of my pull request.




  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message