spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?
Date Mon, 04 Apr 2016 22:37:24 GMT
Thanks, that was the command. :thumbsup:

On Mon, Apr 4, 2016 at 6:28 PM Jakob Odersky <jakob@odersky.com> wrote:

> I just found out how the hash is calculated:
>
> gpg --print-md sha512 <spark-archive>.tgz
>
> you can use that to check if the resulting output matches the contents
> of <spark-archive>.tgz.sha
>
> On Mon, Apr 4, 2016 at 3:19 PM, Jakob Odersky <jakob@odersky.com> wrote:
> > The published hash is a SHA512.
> >
> > You can verify the integrity of the packages by running `sha512sum` on
> > the archive and comparing the computed hash with the published one.
> > Unfortunately however, I don't know what tool is used to generate the
> > hash and I can't reproduce the format, so I ended up manually
> > comparing the hashes.
> >
> > On Mon, Apr 4, 2016 at 2:39 PM, Nicholas Chammas
> > <nicholas.chammas@gmail.com> wrote:
> >> An additional note: The Spark packages being served off of CloudFront
> (i.e.
> >> the “direct download” option on spark.apache.org) are also corrupt.
> >>
> >> Btw what’s the correct way to verify the SHA of a Spark package? I’ve
> tried
> >> a few commands on working packages downloaded from Apache mirrors, but I
> >> can’t seem to reproduce the published SHA for
> spark-1.6.1-bin-hadoop2.6.tgz.
> >>
> >>
> >> On Mon, Apr 4, 2016 at 11:45 AM Ted Yu <yuzhihong@gmail.com> wrote:
> >>>
> >>> Maybe temporarily take out the artifacts on S3 before the root cause is
> >>> found.
> >>>
> >>> On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas
> >>> <nicholas.chammas@gmail.com> wrote:
> >>>>
> >>>> Just checking in on this again as the builds on S3 are still broken.
> :/
> >>>>
> >>>> Could it have something to do with us moving release-build.sh?
> >>>>
> >>>>
> >>>> On Mon, Mar 21, 2016 at 1:43 PM Nicholas Chammas
> >>>> <nicholas.chammas@gmail.com> wrote:
> >>>>>
> >>>>> Is someone going to retry fixing these packages? It's still a
> problem.
> >>>>>
> >>>>> Also, it would be good to understand why this is happening.
> >>>>>
> >>>>> On Fri, Mar 18, 2016 at 6:49 PM Jakob Odersky <jakob@odersky.com>
> wrote:
> >>>>>>
> >>>>>> I just realized you're using a different download site. Sorry
for
> the
> >>>>>> confusion, the link I get for a direct download of Spark 1.6.1
/
> >>>>>> Hadoop 2.6 is
> >>>>>> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz
> >>>>>>
> >>>>>> On Fri, Mar 18, 2016 at 3:20 PM, Nicholas Chammas
> >>>>>> <nicholas.chammas@gmail.com> wrote:
> >>>>>> > I just retried the Spark 1.6.1 / Hadoop 2.6 download and
got a
> >>>>>> > corrupt ZIP
> >>>>>> > file.
> >>>>>> >
> >>>>>> > Jakob, are you sure the ZIP unpacks correctly for you?
Is it the
> same
> >>>>>> > Spark
> >>>>>> > 1.6.1/Hadoop 2.6 package you had a success with?
> >>>>>> >
> >>>>>> > On Fri, Mar 18, 2016 at 6:11 PM Jakob Odersky <jakob@odersky.com>
> >>>>>> > wrote:
> >>>>>> >>
> >>>>>> >> I just experienced the issue, however retrying the
download a
> second
> >>>>>> >> time worked. Could it be that there is some load balancer/cache
> in
> >>>>>> >> front of the archive and some nodes still serve the
corrupt
> >>>>>> >> packages?
> >>>>>> >>
> >>>>>> >> On Fri, Mar 18, 2016 at 8:00 AM, Nicholas Chammas
> >>>>>> >> <nicholas.chammas@gmail.com> wrote:
> >>>>>> >> > I'm seeing the same. :(
> >>>>>> >> >
> >>>>>> >> > On Fri, Mar 18, 2016 at 10:57 AM Ted Yu <yuzhihong@gmail.com>
> >>>>>> >> > wrote:
> >>>>>> >> >>
> >>>>>> >> >> I tried again this morning :
> >>>>>> >> >>
> >>>>>> >> >> $ wget
> >>>>>> >> >>
> >>>>>> >> >>
> >>>>>> >> >>
> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
> >>>>>> >> >> --2016-03-18 07:55:30--
> >>>>>> >> >>
> >>>>>> >> >>
> >>>>>> >> >>
> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
> >>>>>> >> >> Resolving s3.amazonaws.com... 54.231.19.163
> >>>>>> >> >> ...
> >>>>>> >> >> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz
> >>>>>> >> >>
> >>>>>> >> >> gzip: stdin: unexpected end of file
> >>>>>> >> >> tar: Unexpected EOF in archive
> >>>>>> >> >> tar: Unexpected EOF in archive
> >>>>>> >> >> tar: Error is not recoverable: exiting now
> >>>>>> >> >>
> >>>>>> >> >> On Thu, Mar 17, 2016 at 8:57 AM, Michael Armbrust
> >>>>>> >> >> <michael@databricks.com>
> >>>>>> >> >> wrote:
> >>>>>> >> >>>
> >>>>>> >> >>> Patrick reuploaded the artifacts, so it
should be fixed now.
> >>>>>> >> >>>
> >>>>>> >> >>> On Mar 16, 2016 5:48 PM, "Nicholas Chammas"
> >>>>>> >> >>> <nicholas.chammas@gmail.com>
> >>>>>> >> >>> wrote:
> >>>>>> >> >>>>
> >>>>>> >> >>>> Looks like the other packages may
also be corrupt. I’m
> getting
> >>>>>> >> >>>> the
> >>>>>> >> >>>> same
> >>>>>> >> >>>> error for the Spark 1.6.1 / Hadoop
2.4 package.
> >>>>>> >> >>>>
> >>>>>> >> >>>>
> >>>>>> >> >>>>
> >>>>>> >> >>>>
> >>>>>> >> >>>>
> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz
> >>>>>> >> >>>>
> >>>>>> >> >>>> Nick
> >>>>>> >> >>>>
> >>>>>> >> >>>>
> >>>>>> >> >>>> On Wed, Mar 16, 2016 at 8:28 PM Ted
Yu <yuzhihong@gmail.com
> >
> >>>>>> >> >>>> wrote:
> >>>>>> >> >>>>>
> >>>>>> >> >>>>> On Linux, I got:
> >>>>>> >> >>>>>
> >>>>>> >> >>>>> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz
> >>>>>> >> >>>>>
> >>>>>> >> >>>>> gzip: stdin: unexpected end of
file
> >>>>>> >> >>>>> tar: Unexpected EOF in archive
> >>>>>> >> >>>>> tar: Unexpected EOF in archive
> >>>>>> >> >>>>> tar: Error is not recoverable:
exiting now
> >>>>>> >> >>>>>
> >>>>>> >> >>>>> On Wed, Mar 16, 2016 at 5:15 PM,
Nicholas Chammas
> >>>>>> >> >>>>> <nicholas.chammas@gmail.com>
wrote:
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>>
> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>> Does anyone else have trouble
unzipping this? How did this
> >>>>>> >> >>>>>> happen?
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>> What I get is:
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>> $ gzip -t spark-1.6.1-bin-hadoop2.6.tgz
> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz:
unexpected end of
> file
> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz:
uncompress failed
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>> Seems like a strange type
of problem to come across.
> >>>>>> >> >>>>>>
> >>>>>> >> >>>>>> Nick
> >>>>>> >> >>>>>
> >>>>>> >> >>>>>
> >>>>>> >> >>
> >>>>>> >> >
> >>>
> >>>
> >>
>

Mime
View raw message