spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp ☠ <skn...@berkeley.edu>
Subject Re: Running K8s integration tests for changes in core?
Date Thu, 24 Sep 2020 17:17:05 GMT
just revisiting this thread...

re presubmit strategy:  i don't think this would be easy to set up...  and
i'm not sure what benefit it will give us.

re inadvertent errors:  since we're checking out the same hash from the PR
for both builds, and they'll run simultaneously, i don't think it'll be an
issue.

re overloading the workers:  nah.  the regular PRB takes ~4hr, and the k8s
PRB takes ~30m and runs in parallel.

i'll set this up right now and keep an eye on the queue/build results today.

shane

On Thu, Aug 20, 2020 at 2:28 PM Holden Karau <holden@pigscanfly.ca> wrote:

> Sounds good, thanks for the heads up. I hope you get some time to relax :)
>
> On Thu, Aug 20, 2020 at 2:26 PM shane knapp ☠ <sknapp@berkeley.edu> wrote:
>
>> fyi, i won't be making this change until the 1st week of september.  i'll
>> be out, off the grid all next week!  :)
>>
>> i will send an announcement out tomorrow on how to contact my team here @
>> uc berkeley if jenkins goes down.
>>
>> shane
>>
>> On Thu, Aug 20, 2020 at 4:40 AM Prashant Sharma <scrapcodes@gmail.com>
>> wrote:
>>
>>> Another option is, if we could have something like "presubmit" PR build.
>>> In other words, running the entire 4 H + K8s integration on each commit
>>> pushed is too much at the same time and there are chances that one thing
>>> can inadvertently affect other components(as you just said).
>>>
>>> A presubmit(which includes K8s integration tests) build will be run,
>>> once the PR receives LGTM from "Approved reviewers". This is one criteria
>>> that comes to my mind, others may have better suggestions.
>>>
>>> On Thu, Aug 20, 2020 at 12:25 AM shane knapp ☠ <sknapp@berkeley.edu>
>>> wrote:
>>>
>>>> we'll be gated by the number of ubuntu workers w/minikube and docker,
>>>> but it shouldn't be too bad as the full integration test takes ~45m, vs 4+
>>>> hrs for the regular PRB.
>>>>
>>>> i can enable this in about 1m of time if the consensus is for us to
>>>> want this.
>>>>
>>>> On Wed, Aug 19, 2020 at 11:37 AM Holden Karau <holden@pigscanfly.ca>
>>>> wrote:
>>>>
>>>>> Sounds good. In the meantime would folks committing things in core run
>>>>> the K8s PRB or run it locally? A second change this morning was committed
>>>>> that broke the K8s PR tests.
>>>>>
>>>>> On Tue, Aug 18, 2020 at 9:53 PM Prashant Sharma <scrapcodes@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1, we should enable.
>>>>>>
>>>>>> On Wed, Aug 19, 2020 at 9:18 AM Holden Karau <holden@pigscanfly.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Dev Folks,
>>>>>>>
>>>>>>> I was wondering how people feel about enabling the K8s PRB
>>>>>>> automatically for all core changes? Sometimes I forget that a
change might
>>>>>>> impact one of the K8s integration tests since a bunch of them
look at log
>>>>>>> messages. Would folks be OK with turning on the K8s integration
PRB for all
>>>>>>> core changes as well as K8s changes?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Holden :)
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>
>>>>
>>>>
>>>> --
>>>> Shane Knapp
>>>> Computer Guy / Voice of Reason
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>>
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>


-- 
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Mime
View raw message