spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From skip cruse <f8al.er...@gmail.com>
Subject Re: [SPOT-266] ml_ops.sh script should use Spark 2?
Date Mon, 23 Mar 2020 14:51:56 GMT
That seems like a fair solution. I noticed there’s also a spark-submit
command that’s required when running this pipeline:worke
<https://github.com/apache/incubator-spot/blob/9f63ab724e075fc3da458bd8ff86a777ddb94d68/spot-ingest/pipelines/proxy/worker.py>
r.py . We should make the update here as well since I believe this also
requries Spark 2.1+.

On Fri, Mar 20, 2020 at 4:24 PM Tadd Wood <taddwood@apache.org> wrote:

> I notice there’s a dependency on Spark 2.1+ that didn’t get resolved per
> this JIRA (https://issues.apache.org/jira/projects/SPOT/issues/SPOT-266).
> I
> propose adding a check in this script to verify the user’s version of Spark
> and if it's not at least 2.1 the script will exit and indicate that the
> user should upgrade their version of Spark. We also need to replace
> spark-submit with spark-submit2. Any thoughts for/against updating this
> script?
>
> Also would anyone be willing to update the documentation on the Spot
> website to reflect the correct required version of Spark to run Spot?
>
> Thank you,
> Tadd Wood
>


-- 
if( bool halfWayThere == true);
echo "WOOOOOOOOOOOOOOOOAH! LIVIN ON A PRAYER";

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message