spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Re: Please keep s3://spark-related-packages/ alive
Date Fri, 02 Mar 2018 04:18:41 GMT
Marton,

Thanks for the tip. (Too bad the docs
<https://reference.apache.org/pmc/mirror_scripts> referenced from the issue
I opened with INFRA <https://issues.apache.org/jira/browse/INFRA-10999> make
no mention of mirrors.cgi.)

Matei,

A Requester Pays bucket is a good idea. I was trying to avoid having to
maintain a repository of assets, but I suppose it's ultimately unavoidable
given that Apache does not partner with a CDN. I will look into this for
Flintrock.

Nick

On Wed, Feb 28, 2018 at 7:21 AM Marton, Elek <elek@apache.org> wrote:

>
> >  2. *Apache mirrors are inconvenient to use.* When you download
> >     something from an Apache mirror, you get a link like this one
> >     <
> https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
> >.
> >     Instead of automatically redirecting you to your download, though,
> >     you need to process the results you get back
> >     <
> https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42
> >
> >     to find your download target. And you need to handle the high
> >     download failure rate, since sometimes the mirror you get doesn’t
> >     have the file it claims to have.
>
> It's not a full answer, just a note:
>
> You can also use mirrors.cgi instead of parsing the json from closer.lua:
>
>
> https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
>
> (Unfortunatelly it doesn't check the availibility of the file. If it's
> moved to the archive you will be redirected to a 404)
>
> Marton
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message