spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyukjin Kwon <gurwls...@gmail.com>
Subject Re: Running K8s integration tests for changes in core?
Date Thu, 24 Sep 2020 22:55:29 GMT
+1

On Fri, 25 Sep 2020, 02:21 Holden Karau, <holden@pigscanfly.ca> wrote:

> Thanks Shane!
>
> On Thu, Sep 24, 2020 at 10:17 AM shane knapp ☠ <sknapp@berkeley.edu>
> wrote:
>
>> just revisiting this thread...
>>
>> re presubmit strategy:  i don't think this would be easy to set up...
>> and i'm not sure what benefit it will give us.
>>
>> re inadvertent errors:  since we're checking out the same hash from the
>> PR for both builds, and they'll run simultaneously, i don't think it'll be
>> an issue.
>>
>> re overloading the workers:  nah.  the regular PRB takes ~4hr, and the
>> k8s PRB takes ~30m and runs in parallel.
>>
>> i'll set this up right now and keep an eye on the queue/build results
>> today.
>>
>> shane
>>
>> On Thu, Aug 20, 2020 at 2:28 PM Holden Karau <holden@pigscanfly.ca>
>> wrote:
>>
>>> Sounds good, thanks for the heads up. I hope you get some time to relax
>>> :)
>>>
>>> On Thu, Aug 20, 2020 at 2:26 PM shane knapp ☠ <sknapp@berkeley.edu>
>>> wrote:
>>>
>>>> fyi, i won't be making this change until the 1st week of september.
>>>> i'll be out, off the grid all next week!  :)
>>>>
>>>> i will send an announcement out tomorrow on how to contact my team
>>>> here @ uc berkeley if jenkins goes down.
>>>>
>>>> shane
>>>>
>>>> On Thu, Aug 20, 2020 at 4:40 AM Prashant Sharma <scrapcodes@gmail.com>
>>>> wrote:
>>>>
>>>>> Another option is, if we could have something like "presubmit" PR
>>>>> build. In other words, running the entire 4 H + K8s integration on each
>>>>> commit pushed is too much at the same time and there are chances that
one
>>>>> thing can inadvertently affect other components(as you just said).
>>>>>
>>>>> A presubmit(which includes K8s integration tests) build will be run,
>>>>> once the PR receives LGTM from "Approved reviewers". This is one criteria
>>>>> that comes to my mind, others may have better suggestions.
>>>>>
>>>>> On Thu, Aug 20, 2020 at 12:25 AM shane knapp ☠ <sknapp@berkeley.edu>
>>>>> wrote:
>>>>>
>>>>>> we'll be gated by the number of ubuntu workers w/minikube and docker,
>>>>>> but it shouldn't be too bad as the full integration test takes ~45m,
vs 4+
>>>>>> hrs for the regular PRB.
>>>>>>
>>>>>> i can enable this in about 1m of time if the consensus is for us
to
>>>>>> want this.
>>>>>>
>>>>>> On Wed, Aug 19, 2020 at 11:37 AM Holden Karau <holden@pigscanfly.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> Sounds good. In the meantime would folks committing things in
core
>>>>>>> run the K8s PRB or run it locally? A second change this morning
was
>>>>>>> committed that broke the K8s PR tests.
>>>>>>>
>>>>>>> On Tue, Aug 18, 2020 at 9:53 PM Prashant Sharma <
>>>>>>> scrapcodes@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1, we should enable.
>>>>>>>>
>>>>>>>> On Wed, Aug 19, 2020 at 9:18 AM Holden Karau <holden@pigscanfly.ca>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Dev Folks,
>>>>>>>>>
>>>>>>>>> I was wondering how people feel about enabling the K8s
PRB
>>>>>>>>> automatically for all core changes? Sometimes I forget
that a change might
>>>>>>>>> impact one of the K8s integration tests since a bunch
of them look at log
>>>>>>>>> messages. Would folks be OK with turning on the K8s integration
PRB for all
>>>>>>>>> core changes as well as K8s changes?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Holden :)
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shane Knapp
>>>>>> Computer Guy / Voice of Reason
>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>>> https://rise.cs.berkeley.edu
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Shane Knapp
>>>> Computer Guy / Voice of Reason
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Mime
View raw message