spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: [YARN] Small fix for yarn.Client to use buildPath (not Path.SEPARATOR)
Date Wed, 15 Jun 2016 08:49:59 GMT

> On 14 Jun 2016, at 13:58, Jacek Laskowski <jacek@japila.pl> wrote:
> 
> Hi Steve and Sean,
> 
> Didn't expect such a warm welcome from Sean and you! Since I'm with
> Spark on YARN these days, let me see what I can do to make it nicer.
> Thanks!
> 
> I'm going to change Spark to use buildPath first. And then propose
> another patch to use Environment.CLASS_PATH_SEPARATOR instead. And
> only then I could work on
> https://issues.apache.org/jira/browse/YARN-5247. Is this about
> changing the annotation(s) only?
> 

yes; though I should warn that I have evidence that some people have been breaking things
tagged a stable; 

https://issues.apache.org/jira/browse/YARN-5130

what that @Stable marker does do is give you ability to complain when things break

> Thanks for your support!
> 
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
> 
> 
> On Tue, Jun 14, 2016 at 1:44 PM, Steve Loughran <stevel@hortonworks.com> wrote:
>> 
>> if you want to be able to build up CPs on windows to run on a Linux cluster, or vice-versa,
you really need to be using the Environment.CLASS_PATH_SEPARATOR field, "<CPS>". This
is expanded in the cluster, not in the client
>> 
>> Although tagged as @Public, @Unstable, it's been in there sinceYARN-1824 & Hadoop
2.4; things rely on it. If someone wants to fix that by submitting a patch to YARN-5247; I'll
review it.
>> 
>>> On 13 Jun 2016, at 20:06, Sean Owen <sowen@cloudera.com> wrote:
>>> 
>>> Yeah it does the same thing anyway. It's fine to consistently use the
>>> method. I think there's an instance in ClientSuite that can use it.
>>> 
>>> On Mon, Jun 13, 2016 at 6:50 PM, Jacek Laskowski <jacek@japila.pl> wrote:
>>>> Hi,
>>>> 
>>>> Just noticed that yarn.Client#populateClasspath uses Path.SEPARATOR
>>>> [1] to build a CLASSPATH entry while another similar-looking line uses
>>>> buildPath method [2].
>>>> 
>>>> Could a pull request with a change to use buildPath at [1] be
>>>> accepted? I'm always confused how to fix such small changes.
>>>> 
>>>> [1] https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1298
>>>> [2] Path.SEPARATOR
>>>> 
>>>> Pozdrawiam,
>>>> Jacek Laskowski
>>>> ----
>>>> https://medium.com/@jaceklaskowski/
>>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>>>> Follow me at https://twitter.com/jaceklaskowski
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>> 
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message