spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Rosen <joshro...@databricks.com>
Subject Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?
Date Wed, 06 Apr 2016 19:19:11 GMT
I downloaded the Spark 1.6.1 artifacts from the Apache mirror network and
re-uploaded them to the spark-related-packages S3 bucket, so hopefully
these packages should be fixed now.

On Mon, Apr 4, 2016 at 3:37 PM Nicholas Chammas <nicholas.chammas@gmail.com>
wrote:

> Thanks, that was the command. :thumbsup:
>
> On Mon, Apr 4, 2016 at 6:28 PM Jakob Odersky <jakob@odersky.com> wrote:
>
>> I just found out how the hash is calculated:
>>
>> gpg --print-md sha512 <spark-archive>.tgz
>>
>> you can use that to check if the resulting output matches the contents
>> of <spark-archive>.tgz.sha
>>
>> On Mon, Apr 4, 2016 at 3:19 PM, Jakob Odersky <jakob@odersky.com> wrote:
>> > The published hash is a SHA512.
>> >
>> > You can verify the integrity of the packages by running `sha512sum` on
>> > the archive and comparing the computed hash with the published one.
>> > Unfortunately however, I don't know what tool is used to generate the
>> > hash and I can't reproduce the format, so I ended up manually
>> > comparing the hashes.
>> >
>> > On Mon, Apr 4, 2016 at 2:39 PM, Nicholas Chammas
>> > <nicholas.chammas@gmail.com> wrote:
>> >> An additional note: The Spark packages being served off of CloudFront
>> (i.e.
>> >> the “direct download” option on spark.apache.org) are also corrupt.
>> >>
>> >> Btw what’s the correct way to verify the SHA of a Spark package? I’ve
>> tried
>> >> a few commands on working packages downloaded from Apache mirrors, but
>> I
>> >> can’t seem to reproduce the published SHA for
>> spark-1.6.1-bin-hadoop2.6.tgz.
>> >>
>> >>
>> >> On Mon, Apr 4, 2016 at 11:45 AM Ted Yu <yuzhihong@gmail.com> wrote:
>> >>>
>> >>> Maybe temporarily take out the artifacts on S3 before the root cause
>> is
>> >>> found.
>> >>>
>> >>> On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas
>> >>> <nicholas.chammas@gmail.com> wrote:
>> >>>>
>> >>>> Just checking in on this again as the builds on S3 are still broken.
>> :/
>> >>>>
>> >>>> Could it have something to do with us moving release-build.sh?
>> >>>>
>> >>>>
>> >>>> On Mon, Mar 21, 2016 at 1:43 PM Nicholas Chammas
>> >>>> <nicholas.chammas@gmail.com> wrote:
>> >>>>>
>> >>>>> Is someone going to retry fixing these packages? It's still
a
>> problem.
>> >>>>>
>> >>>>> Also, it would be good to understand why this is happening.
>> >>>>>
>> >>>>> On Fri, Mar 18, 2016 at 6:49 PM Jakob Odersky <jakob@odersky.com>
>> wrote:
>> >>>>>>
>> >>>>>> I just realized you're using a different download site.
Sorry for
>> the
>> >>>>>> confusion, the link I get for a direct download of Spark
1.6.1 /
>> >>>>>> Hadoop 2.6 is
>> >>>>>> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz
>> >>>>>>
>> >>>>>> On Fri, Mar 18, 2016 at 3:20 PM, Nicholas Chammas
>> >>>>>> <nicholas.chammas@gmail.com> wrote:
>> >>>>>> > I just retried the Spark 1.6.1 / Hadoop 2.6 download
and got a
>> >>>>>> > corrupt ZIP
>> >>>>>> > file.
>> >>>>>> >
>> >>>>>> > Jakob, are you sure the ZIP unpacks correctly for you?
Is it the
>> same
>> >>>>>> > Spark
>> >>>>>> > 1.6.1/Hadoop 2.6 package you had a success with?
>> >>>>>> >
>> >>>>>> > On Fri, Mar 18, 2016 at 6:11 PM Jakob Odersky <jakob@odersky.com
>> >
>> >>>>>> > wrote:
>> >>>>>> >>
>> >>>>>> >> I just experienced the issue, however retrying
the download a
>> second
>> >>>>>> >> time worked. Could it be that there is some load
balancer/cache
>> in
>> >>>>>> >> front of the archive and some nodes still serve
the corrupt
>> >>>>>> >> packages?
>> >>>>>> >>
>> >>>>>> >> On Fri, Mar 18, 2016 at 8:00 AM, Nicholas Chammas
>> >>>>>> >> <nicholas.chammas@gmail.com> wrote:
>> >>>>>> >> > I'm seeing the same. :(
>> >>>>>> >> >
>> >>>>>> >> > On Fri, Mar 18, 2016 at 10:57 AM Ted Yu <yuzhihong@gmail.com>
>> >>>>>> >> > wrote:
>> >>>>>> >> >>
>> >>>>>> >> >> I tried again this morning :
>> >>>>>> >> >>
>> >>>>>> >> >> $ wget
>> >>>>>> >> >>
>> >>>>>> >> >>
>> >>>>>> >> >>
>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
>> >>>>>> >> >> --2016-03-18 07:55:30--
>> >>>>>> >> >>
>> >>>>>> >> >>
>> >>>>>> >> >>
>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
>> >>>>>> >> >> Resolving s3.amazonaws.com... 54.231.19.163
>> >>>>>> >> >> ...
>> >>>>>> >> >> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz
>> >>>>>> >> >>
>> >>>>>> >> >> gzip: stdin: unexpected end of file
>> >>>>>> >> >> tar: Unexpected EOF in archive
>> >>>>>> >> >> tar: Unexpected EOF in archive
>> >>>>>> >> >> tar: Error is not recoverable: exiting
now
>> >>>>>> >> >>
>> >>>>>> >> >> On Thu, Mar 17, 2016 at 8:57 AM, Michael
Armbrust
>> >>>>>> >> >> <michael@databricks.com>
>> >>>>>> >> >> wrote:
>> >>>>>> >> >>>
>> >>>>>> >> >>> Patrick reuploaded the artifacts,
so it should be fixed now.
>> >>>>>> >> >>>
>> >>>>>> >> >>> On Mar 16, 2016 5:48 PM, "Nicholas
Chammas"
>> >>>>>> >> >>> <nicholas.chammas@gmail.com>
>> >>>>>> >> >>> wrote:
>> >>>>>> >> >>>>
>> >>>>>> >> >>>> Looks like the other packages
may also be corrupt. I’m
>> getting
>> >>>>>> >> >>>> the
>> >>>>>> >> >>>> same
>> >>>>>> >> >>>> error for the Spark 1.6.1 / Hadoop
2.4 package.
>> >>>>>> >> >>>>
>> >>>>>> >> >>>>
>> >>>>>> >> >>>>
>> >>>>>> >> >>>>
>> >>>>>> >> >>>>
>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz
>> >>>>>> >> >>>>
>> >>>>>> >> >>>> Nick
>> >>>>>> >> >>>>
>> >>>>>> >> >>>>
>> >>>>>> >> >>>> On Wed, Mar 16, 2016 at 8:28 PM
Ted Yu <
>> yuzhihong@gmail.com>
>> >>>>>> >> >>>> wrote:
>> >>>>>> >> >>>>>
>> >>>>>> >> >>>>> On Linux, I got:
>> >>>>>> >> >>>>>
>> >>>>>> >> >>>>> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz
>> >>>>>> >> >>>>>
>> >>>>>> >> >>>>> gzip: stdin: unexpected end
of file
>> >>>>>> >> >>>>> tar: Unexpected EOF in archive
>> >>>>>> >> >>>>> tar: Unexpected EOF in archive
>> >>>>>> >> >>>>> tar: Error is not recoverable:
exiting now
>> >>>>>> >> >>>>>
>> >>>>>> >> >>>>> On Wed, Mar 16, 2016 at 5:15
PM, Nicholas Chammas
>> >>>>>> >> >>>>> <nicholas.chammas@gmail.com>
wrote:
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>>
>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>> Does anyone else have
trouble unzipping this? How did
>> this
>> >>>>>> >> >>>>>> happen?
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>> What I get is:
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>> $ gzip -t spark-1.6.1-bin-hadoop2.6.tgz
>> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz:
unexpected end of
>> file
>> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz:
uncompress failed
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>> Seems like a strange type
of problem to come across.
>> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>> Nick
>> >>>>>> >> >>>>>
>> >>>>>> >> >>>>>
>> >>>>>> >> >>
>> >>>>>> >> >
>> >>>
>> >>>
>> >>
>>
>

Mime
View raw message