spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Odersky <ja...@odersky.com>
Subject Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?
Date Mon, 04 Apr 2016 22:19:48 GMT
The published hash is a SHA512.

You can verify the integrity of the packages by running `sha512sum` on
the archive and comparing the computed hash with the published one.
Unfortunately however, I don't know what tool is used to generate the
hash and I can't reproduce the format, so I ended up manually
comparing the hashes.

On Mon, Apr 4, 2016 at 2:39 PM, Nicholas Chammas
<nicholas.chammas@gmail.com> wrote:
> An additional note: The Spark packages being served off of CloudFront (i.e.
> the “direct download” option on spark.apache.org) are also corrupt.
>
> Btw what’s the correct way to verify the SHA of a Spark package? I’ve tried
> a few commands on working packages downloaded from Apache mirrors, but I
> can’t seem to reproduce the published SHA for spark-1.6.1-bin-hadoop2.6.tgz.
>
>
> On Mon, Apr 4, 2016 at 11:45 AM Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> Maybe temporarily take out the artifacts on S3 before the root cause is
>> found.
>>
>> On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas
>> <nicholas.chammas@gmail.com> wrote:
>>>
>>> Just checking in on this again as the builds on S3 are still broken. :/
>>>
>>> Could it have something to do with us moving release-build.sh?
>>>
>>>
>>> On Mon, Mar 21, 2016 at 1:43 PM Nicholas Chammas
>>> <nicholas.chammas@gmail.com> wrote:
>>>>
>>>> Is someone going to retry fixing these packages? It's still a problem.
>>>>
>>>> Also, it would be good to understand why this is happening.
>>>>
>>>> On Fri, Mar 18, 2016 at 6:49 PM Jakob Odersky <jakob@odersky.com> wrote:
>>>>>
>>>>> I just realized you're using a different download site. Sorry for the
>>>>> confusion, the link I get for a direct download of Spark 1.6.1 /
>>>>> Hadoop 2.6 is
>>>>> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz
>>>>>
>>>>> On Fri, Mar 18, 2016 at 3:20 PM, Nicholas Chammas
>>>>> <nicholas.chammas@gmail.com> wrote:
>>>>> > I just retried the Spark 1.6.1 / Hadoop 2.6 download and got a
>>>>> > corrupt ZIP
>>>>> > file.
>>>>> >
>>>>> > Jakob, are you sure the ZIP unpacks correctly for you? Is it the
same
>>>>> > Spark
>>>>> > 1.6.1/Hadoop 2.6 package you had a success with?
>>>>> >
>>>>> > On Fri, Mar 18, 2016 at 6:11 PM Jakob Odersky <jakob@odersky.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> I just experienced the issue, however retrying the download
a second
>>>>> >> time worked. Could it be that there is some load balancer/cache
in
>>>>> >> front of the archive and some nodes still serve the corrupt
>>>>> >> packages?
>>>>> >>
>>>>> >> On Fri, Mar 18, 2016 at 8:00 AM, Nicholas Chammas
>>>>> >> <nicholas.chammas@gmail.com> wrote:
>>>>> >> > I'm seeing the same. :(
>>>>> >> >
>>>>> >> > On Fri, Mar 18, 2016 at 10:57 AM Ted Yu <yuzhihong@gmail.com>
>>>>> >> > wrote:
>>>>> >> >>
>>>>> >> >> I tried again this morning :
>>>>> >> >>
>>>>> >> >> $ wget
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
>>>>> >> >> --2016-03-18 07:55:30--
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
>>>>> >> >> Resolving s3.amazonaws.com... 54.231.19.163
>>>>> >> >> ...
>>>>> >> >> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz
>>>>> >> >>
>>>>> >> >> gzip: stdin: unexpected end of file
>>>>> >> >> tar: Unexpected EOF in archive
>>>>> >> >> tar: Unexpected EOF in archive
>>>>> >> >> tar: Error is not recoverable: exiting now
>>>>> >> >>
>>>>> >> >> On Thu, Mar 17, 2016 at 8:57 AM, Michael Armbrust
>>>>> >> >> <michael@databricks.com>
>>>>> >> >> wrote:
>>>>> >> >>>
>>>>> >> >>> Patrick reuploaded the artifacts, so it should
be fixed now.
>>>>> >> >>>
>>>>> >> >>> On Mar 16, 2016 5:48 PM, "Nicholas Chammas"
>>>>> >> >>> <nicholas.chammas@gmail.com>
>>>>> >> >>> wrote:
>>>>> >> >>>>
>>>>> >> >>>> Looks like the other packages may also be corrupt.
I’m getting
>>>>> >> >>>> the
>>>>> >> >>>> same
>>>>> >> >>>> error for the Spark 1.6.1 / Hadoop 2.4 package.
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz
>>>>> >> >>>>
>>>>> >> >>>> Nick
>>>>> >> >>>>
>>>>> >> >>>>
>>>>> >> >>>> On Wed, Mar 16, 2016 at 8:28 PM Ted Yu <yuzhihong@gmail.com>
>>>>> >> >>>> wrote:
>>>>> >> >>>>>
>>>>> >> >>>>> On Linux, I got:
>>>>> >> >>>>>
>>>>> >> >>>>> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz
>>>>> >> >>>>>
>>>>> >> >>>>> gzip: stdin: unexpected end of file
>>>>> >> >>>>> tar: Unexpected EOF in archive
>>>>> >> >>>>> tar: Unexpected EOF in archive
>>>>> >> >>>>> tar: Error is not recoverable: exiting
now
>>>>> >> >>>>>
>>>>> >> >>>>> On Wed, Mar 16, 2016 at 5:15 PM, Nicholas
Chammas
>>>>> >> >>>>> <nicholas.chammas@gmail.com> wrote:
>>>>> >> >>>>>>
>>>>> >> >>>>>>
>>>>> >> >>>>>>
>>>>> >> >>>>>>
>>>>> >> >>>>>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
>>>>> >> >>>>>>
>>>>> >> >>>>>> Does anyone else have trouble unzipping
this? How did this
>>>>> >> >>>>>> happen?
>>>>> >> >>>>>>
>>>>> >> >>>>>> What I get is:
>>>>> >> >>>>>>
>>>>> >> >>>>>> $ gzip -t spark-1.6.1-bin-hadoop2.6.tgz
>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz:
unexpected end of file
>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz:
uncompress failed
>>>>> >> >>>>>>
>>>>> >> >>>>>> Seems like a strange type of problem
to come across.
>>>>> >> >>>>>>
>>>>> >> >>>>>> Nick
>>>>> >> >>>>>
>>>>> >> >>>>>
>>>>> >> >>
>>>>> >> >
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message