spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <dbt...@dbtsai.com.INVALID>
Subject Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Date Sun, 10 Mar 2019 22:42:09 GMT
As we have many important fixes in 2.4 branch which we want to release
asap, and this is is not a regression from Spark 2.4; as a result, 2.4.1
will be not blocked by this.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0359BC9965359766


On Sun, Mar 10, 2019 at 3:08 PM Michael Heuer <heuermh@gmail.com> wrote:

> Any chance we could get some movement on this for 2.4.1?
>
> https://issues.apache.org/jira/browse/SPARK-25588
> https://github.com/apache/parquet-mr/pull/560
>
> It would require a new Parquet release, which would then need to be picked
> up by Spark.  We're dead in the water on 2.4.0 without a large refactoring
> (remove all the RDD code paths for reading Avro stored in Parquet).
>
>    michael
>
>
> On Mar 8, 2019, at 6:22 PM, Sean Owen <srowen@gmail.com> wrote:
>
> FWIW RC6 looked fine to me. Passed all tests, etc.
>
> On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <dbtsai@dbtsai.com> wrote:
>
>> Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully,
>> this is the final rc. Thanks.
>>
>> Sincerely,
>>
>> DB Tsai
>> ----------------------------------------------------------
>> Web: https://www.dbtsai.com
>> PGP Key ID: 42E5B25A8F7A82C1
>>
>>
>> On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <lixiao@databricks.com> wrote:
>>
>>> It is common to hit this issue when driver and executors are different
>>> object layout, but Spark might not return a wrong answer. It is very hard
>>> to find out the root cause. Thus, I would suggest to include it in Spark
>>> 2.4.1.
>>>
>>> On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <dbtsai@dbtsai.com> wrote:
>>>
>>>> BTW, practically, is it common for users running into this bug when the
>>>> driver and executors have different object layout?
>>>>
>>>> Sincerely,
>>>>
>>>> DB Tsai
>>>> ----------------------------------------------------------
>>>> Web: https://www.dbtsai.com
>>>> PGP Key ID: 42E5B25A8F7A82C1
>>>>
>>>>
>>>> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <dbtsai@dbtsai.com> wrote:
>>>>
>>>>> Hi Xiao,
>>>>>
>>>>> I already cut rc7 and start the build process. If we definitely need
>>>>> this fix, I can cut rc8. Let me know what you think.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <lixiao@databricks.com>
wrote:
>>>>>
>>>>>> Hi, DB,
>>>>>>
>>>>>> Since this RC will fail, could you hold it until we fix
>>>>>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or
I
>>>>>> will submit a PR today. The PR is small and the risk is low. This
is a
>>>>>> correctness bug. It would be good to have it.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Xiao
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_tsai@apple.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> Since I can not find the commit of `Preparing development version
>>>>>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch
and do a
>>>>>>> force-push. I'll follow Marcelo's suggestion to have another
rc7 cut. Thus,
>>>>>>> this vote fails.
>>>>>>>
>>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]
 |  
>>>>>>> Apple, Inc
>>>>>>>
>>>>>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_tsai@apple.com.INVALID>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch.
It's
>>>>>>> very weird. It must be overwritten by a force push.
>>>>>>> >
>>>>>>> > DB Tsai  |  Siri Open Source Technologies [not a contribution]
 |
>>>>>>>  Apple, Inc
>>>>>>> >
>>>>>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_tsai@apple.com.INVALID>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> I was using `./do-release-docker.sh` to create a release.
But
>>>>>>> since the gpg validation fails couple times when the script tried
to
>>>>>>> publish the jars into Nexus, I re-ran the scripts multiple times
without
>>>>>>> creating a new rc. I was wondering if the script will overwrite
the
>>>>>>> v.2.4.1-rc6 tag instead of using the same commit causing this
issue.
>>>>>>> >>
>>>>>>> >> Should we create a new rc7?
>>>>>>> >>
>>>>>>> >> DB Tsai  |  Siri Open Source Technologies [not a contribution]
>>>>>>> |   Apple, Inc
>>>>>>> >>
>>>>>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <
>>>>>>> vanzin@cloudera.com.INVALID> wrote:
>>>>>>> >>>
>>>>>>> >>> I personally find it a little weird to not have
the commit in
>>>>>>> branch-2.4.
>>>>>>> >>>
>>>>>>> >>> Not that this would happen, but if the v2.4.1-rc6
tag is
>>>>>>> overwritten
>>>>>>> >>> (e.g. accidentally) then you lose the reference
to that commit,
>>>>>>> and
>>>>>>> >>> then the exact commit from which the rc was generated
is lost.
>>>>>>> >>>
>>>>>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <srowen@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> That's weird. I see the commit but can't find
it in the branch.
>>>>>>> Was it pushed, or lost in a force push of 2.4 along the way?
The change is
>>>>>>> there, just under a different commit in the 2.4 branch.
>>>>>>> >>>>
>>>>>>> >>>> It doesn't necessarily invalidate the RC as
it is a valid
>>>>>>> public tagged commit and all that. I just want to be sure we
do have the
>>>>>>> code from that commit in these tatballs. It looks like it.
>>>>>>> >>>>
>>>>>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <misutoth@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Hi,
>>>>>>> >>>>>
>>>>>>> >>>>> I am not sure how problematic it is but
v2.4.1-rc6 is not on
>>>>>>> branch-2.4. Release related commits I have seen so far were also
part of
>>>>>>> the branch.
>>>>>>> >>>>>
>>>>>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6"
and
>>>>>>> "Preparing development version 2.4.2-SNAPSHOT" commits were simply
not
>>>>>>> pushed to spark-2.4 just the tag itself was pushed. I dont know
what is the
>>>>>>> practice in such cases but one solution is to rebase branch-2.4
changes
>>>>>>> after 3336a21 onto these commits and do a (sorry) force push.
In this case
>>>>>>> there is no impact on this RC.
>>>>>>> >>>>>
>>>>>>> >>>>> Best Regards,
>>>>>>> >>>>>
>>>>>>> >>>>> Misi
>>>>>>> >>>>>
>>>>>>> >>>>> DB Tsai <d_tsai@apple.com.invalid>
ezt írta (időpont: 2019.
>>>>>>> márc. 8., P, 1:15):
>>>>>>> >>>>>>
>>>>>>> >>>>>> Please vote on releasing the following
candidate as Apache
>>>>>>> Spark version 2.4.1.
>>>>>>> >>>>>>
>>>>>>> >>>>>> The vote is open until March 11 PST
and passes if a majority
>>>>>>> +1 PMC votes are cast, with
>>>>>>> >>>>>> a minimum of 3 +1 votes.
>>>>>>> >>>>>>
>>>>>>> >>>>>> [ ] +1 Release this package as Apache
Spark 2.4.1
>>>>>>> >>>>>> [ ] -1 Do not release this package because
...
>>>>>>> >>>>>>
>>>>>>> >>>>>> To learn more about Apache Spark, please
see
>>>>>>> http://spark.apache.org/
>>>>>>> >>>>>>
>>>>>>> >>>>>> The tag to be voted on is v2.4.1-rc6
(commit
>>>>>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>> >>>>>>
>>>>>>> >>>>>> The release files, including signatures,
digests, etc. can be
>>>>>>> found at:
>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>> >>>>>>
>>>>>>> >>>>>> Signatures used for Spark RCs can be
found in this file:
>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>> >>>>>>
>>>>>>> >>>>>> The staging repository for this release
can be found at:
>>>>>>> >>>>>>
>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>> >>>>>>
>>>>>>> >>>>>> The documentation corresponding to this
release can be found
>>>>>>> at:
>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>> >>>>>>
>>>>>>> >>>>>> The list of bug fixes going into 2.4.1
can be found at the
>>>>>>> following URL:
>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>> >>>>>>
>>>>>>> >>>>>> FAQ
>>>>>>> >>>>>>
>>>>>>> >>>>>> =========================
>>>>>>> >>>>>> How can I help test this release?
>>>>>>> >>>>>> =========================
>>>>>>> >>>>>>
>>>>>>> >>>>>> If you are a Spark user, you can help
us test this release by
>>>>>>> taking
>>>>>>> >>>>>> an existing Spark workload and running
on this release
>>>>>>> candidate, then
>>>>>>> >>>>>> reporting any regressions.
>>>>>>> >>>>>>
>>>>>>> >>>>>> If you're working in PySpark you can
set up a virtual env and
>>>>>>> install
>>>>>>> >>>>>> the current RC and see if anything important
breaks, in the
>>>>>>> Java/Scala
>>>>>>> >>>>>> you can add the staging repository to
your projects resolvers
>>>>>>> and test
>>>>>>> >>>>>> with the RC (make sure to clean up the
artifact cache
>>>>>>> before/after so
>>>>>>> >>>>>> you don't end up building with a out
of date RC going
>>>>>>> forward).
>>>>>>> >>>>>>
>>>>>>> >>>>>> ===========================================
>>>>>>> >>>>>> What should happen to JIRA tickets still
targeting 2.4.1?
>>>>>>> >>>>>> ===========================================
>>>>>>> >>>>>>
>>>>>>> >>>>>> The current list of open tickets targeted
at 2.4.1 can be
>>>>>>> found at:
>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK
and search for
>>>>>>> "Target Version/s" = 2.4.1
>>>>>>> >>>>>>
>>>>>>> >>>>>> Committers should look at those and
triage. Extremely
>>>>>>> important bug
>>>>>>> >>>>>> fixes, documentation, and API tweaks
that impact
>>>>>>> compatibility should
>>>>>>> >>>>>> be worked on immediately. Everything
else please retarget to
>>>>>>> an
>>>>>>> >>>>>> appropriate release.
>>>>>>> >>>>>>
>>>>>>> >>>>>> ==================
>>>>>>> >>>>>> But my bug isn't fixed?
>>>>>>> >>>>>> ==================
>>>>>>> >>>>>>
>>>>>>> >>>>>> In order to make timely releases, we
will typically not hold
>>>>>>> the
>>>>>>> >>>>>> release unless the bug in question is
a regression from the
>>>>>>> previous
>>>>>>> >>>>>> release. That being said, if there is
something which is a
>>>>>>> regression
>>>>>>> >>>>>> that has not been correctly targeted
please ping me or a
>>>>>>> committer to
>>>>>>> >>>>>> help target the issue.
>>>>>>> >>>>>>
>>>>>>> >>>>>> DB Tsai  |  Siri Open Source Technologies
[not a
>>>>>>> contribution]  |   Apple, Inc
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >>>>>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> --
>>>>>>> >>> Marcelo
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >>
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> <SAIS19-emailsig-keynotes-1@2X.jpg>
>>>>>>
>>>>> --
>>>>> - DB Sent from my iPhone
>>>>>
>>>>
>>>
>>> --
>>> <SAIS19-emailsig-keynotes-1@2X.jpg>
>>>
>>
>

Mime
View raw message