spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: [YARN] Small fix for yarn.Client to use buildPath (not Path.SEPARATOR)
Date Tue, 14 Jun 2016 12:58:32 GMT
Hi Steve and Sean,

Didn't expect such a warm welcome from Sean and you! Since I'm with
Spark on YARN these days, let me see what I can do to make it nicer.
Thanks!

I'm going to change Spark to use buildPath first. And then propose
another patch to use Environment.CLASS_PATH_SEPARATOR instead. And
only then I could work on
https://issues.apache.org/jira/browse/YARN-5247. Is this about
changing the annotation(s) only?

Thanks for your support!

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Jun 14, 2016 at 1:44 PM, Steve Loughran <stevel@hortonworks.com> wrote:
>
> if you want to be able to build up CPs on windows to run on a Linux cluster, or vice-versa,
you really need to be using the Environment.CLASS_PATH_SEPARATOR field, "<CPS>". This
is expanded in the cluster, not in the client
>
> Although tagged as @Public, @Unstable, it's been in there sinceYARN-1824 & Hadoop
2.4; things rely on it. If someone wants to fix that by submitting a patch to YARN-5247; I'll
review it.
>
>> On 13 Jun 2016, at 20:06, Sean Owen <sowen@cloudera.com> wrote:
>>
>> Yeah it does the same thing anyway. It's fine to consistently use the
>> method. I think there's an instance in ClientSuite that can use it.
>>
>> On Mon, Jun 13, 2016 at 6:50 PM, Jacek Laskowski <jacek@japila.pl> wrote:
>>> Hi,
>>>
>>> Just noticed that yarn.Client#populateClasspath uses Path.SEPARATOR
>>> [1] to build a CLASSPATH entry while another similar-looking line uses
>>> buildPath method [2].
>>>
>>> Could a pull request with a change to use buildPath at [1] be
>>> accepted? I'm always confused how to fix such small changes.
>>>
>>> [1] https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1298
>>> [2] Path.SEPARATOR
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://medium.com/@jaceklaskowski/
>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message