spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Re: Changing how we compute release hashes
Date Fri, 16 Mar 2018 21:00:47 GMT
OK, will do.

On Fri, Mar 16, 2018 at 4:41 PM Sean Owen <srowen@gmail.com> wrote:

> I think you can file a JIRA and open a PR. All of the bits that use "gpg
> ... SHA512 file ..." can use shasum instead.
> I would not change any existing release artifacts though.
>
> On Fri, Mar 16, 2018 at 1:14 PM Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I have sha512sum on my Mac via Homebrew, but yeah as long as the format
>> is the same I suppose it doesn’t matter if we use shasum -a or sha512sum.
>>
>> So shall I file a JIRA + PR for this? Or should I leave the PR to a
>> maintainer? And are we OK with updating all the existing release hashes to
>> use the new format, or do we only want to do this for new releases?
>> ​
>>
>> On Fri, Mar 16, 2018 at 1:50 PM Felix Cheung <felixcheung_m@hotmail.com>
>> wrote:
>>
>>> +1 there
>>>
>>> ------------------------------
>>> *From:* Sean Owen <srowen@gmail.com>
>>> *Sent:* Friday, March 16, 2018 9:51:49 AM
>>> *To:* Felix Cheung
>>> *Cc:* rblue@netflix.com; Nicholas Chammas; Spark dev list
>>>
>>> *Subject:* Re: Changing how we compute release hashes
>>> I think the issue with that is that OS X doesn't have "sha512sum". Both
>>> it and Linux have "shasum -a 512" though.
>>>
>>> On Fri, Mar 16, 2018 at 11:05 AM Felix Cheung <felixcheung_m@hotmail.com>
>>> wrote:
>>>
>>>> Instead of using gpg to create the sha512 hash file we could just
>>>> change to using sha512sum? That would output the right format that is in
>>>> turns verifiable.
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Ryan Blue <rblue@netflix.com.INVALID>
>>>> *Sent:* Friday, March 16, 2018 8:31:45 AM
>>>> *To:* Nicholas Chammas
>>>> *Cc:* Spark dev list
>>>> *Subject:* Re: Changing how we compute release hashes
>>>>
>>>> +1 It's possible to produce the same file with gpg, but the sha*sum
>>>> utilities are a bit easier to remember the syntax for.
>>>>
>>>> On Thu, Mar 15, 2018 at 9:01 PM, Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>>> To verify that I’ve downloaded a Hadoop release correctly, I can just
>>>>> do this:
>>>>>
>>>>> $ shasum --check hadoop-2.7.5.tar.gz.sha256
>>>>> hadoop-2.7.5.tar.gz: OK
>>>>>
>>>>> However, since we generate Spark release hashes with GPG
>>>>> <https://github.com/apache/spark/blob/c2632edebd978716dbfa7874a2fc0a8f5a4a9951/dev/create-release/release-build.sh#L167-L168>,
>>>>> the resulting hash is in a format that doesn’t play well with any tools:
>>>>>
>>>>> $ shasum --check spark-2.3.0-bin-hadoop2.7.tgz.sha512
>>>>> shasum: spark-2.3.0-bin-hadoop2.7.tgz.sha512: no properly formatted SHA1
checksum lines found
>>>>>
>>>>> GPG doesn’t seem to offer a way to verify a file from a hash.
>>>>>
>>>>> I know I can always manipulate the SHA512 hash into a different format
>>>>> or just manually inspect it, but as a “quality of life” improvement
can we
>>>>> change how we generate the SHA512 hash so that it plays nicely with
>>>>> shasum? If it’s too disruptive to change the format of the SHA512
>>>>> hash, can we add a SHA256 hash to our releases in this format?
>>>>>
>>>>> I suppose if it’s not easy to update or add hashes to our existing
>>>>> releases, it may be too difficult to change anything here. But I’m
not
>>>>> sure, so I thought I’d ask.
>>>>>
>>>>> Nick
>>>>> ​
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>

Mime
View raw message