spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From skip cruse <>
Subject Re: [SPOT-266] script should use Spark 2?
Date Mon, 23 Mar 2020 14:51:56 GMT
That seems like a fair solution. I noticed there’s also a spark-submit
command that’s required when running this pipeline:worke
<> . We should make the update here as well since I believe this also
requries Spark 2.1+.

On Fri, Mar 20, 2020 at 4:24 PM Tadd Wood <> wrote:

> I notice there’s a dependency on Spark 2.1+ that didn’t get resolved per
> this JIRA (
> I
> propose adding a check in this script to verify the user’s version of Spark
> and if it's not at least 2.1 the script will exit and indicate that the
> user should upgrade their version of Spark. We also need to replace
> spark-submit with spark-submit2. Any thoughts for/against updating this
> script?
> Also would anyone be willing to update the documentation on the Spot
> website to reflect the correct required version of Spark to run Spot?
> Thank you,
> Tadd Wood

if( bool halfWayThere == true);

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message