spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Running K8s integration tests for changes in core?
Date Thu, 24 Sep 2020 17:21:27 GMT
Thanks Shane!

On Thu, Sep 24, 2020 at 10:17 AM shane knapp ☠ <sknapp@berkeley.edu> wrote:

> just revisiting this thread...
>
> re presubmit strategy:  i don't think this would be easy to set up...  and
> i'm not sure what benefit it will give us.
>
> re inadvertent errors:  since we're checking out the same hash from the PR
> for both builds, and they'll run simultaneously, i don't think it'll be an
> issue.
>
> re overloading the workers:  nah.  the regular PRB takes ~4hr, and the k8s
> PRB takes ~30m and runs in parallel.
>
> i'll set this up right now and keep an eye on the queue/build results
> today.
>
> shane
>
> On Thu, Aug 20, 2020 at 2:28 PM Holden Karau <holden@pigscanfly.ca> wrote:
>
>> Sounds good, thanks for the heads up. I hope you get some time to relax :)
>>
>> On Thu, Aug 20, 2020 at 2:26 PM shane knapp ☠ <sknapp@berkeley.edu>
>> wrote:
>>
>>> fyi, i won't be making this change until the 1st week of september.
>>> i'll be out, off the grid all next week!  :)
>>>
>>> i will send an announcement out tomorrow on how to contact my team
>>> here @ uc berkeley if jenkins goes down.
>>>
>>> shane
>>>
>>> On Thu, Aug 20, 2020 at 4:40 AM Prashant Sharma <scrapcodes@gmail.com>
>>> wrote:
>>>
>>>> Another option is, if we could have something like "presubmit" PR
>>>> build. In other words, running the entire 4 H + K8s integration on each
>>>> commit pushed is too much at the same time and there are chances that one
>>>> thing can inadvertently affect other components(as you just said).
>>>>
>>>> A presubmit(which includes K8s integration tests) build will be run,
>>>> once the PR receives LGTM from "Approved reviewers". This is one criteria
>>>> that comes to my mind, others may have better suggestions.
>>>>
>>>> On Thu, Aug 20, 2020 at 12:25 AM shane knapp ☠ <sknapp@berkeley.edu>
>>>> wrote:
>>>>
>>>>> we'll be gated by the number of ubuntu workers w/minikube and docker,
>>>>> but it shouldn't be too bad as the full integration test takes ~45m,
vs 4+
>>>>> hrs for the regular PRB.
>>>>>
>>>>> i can enable this in about 1m of time if the consensus is for us to
>>>>> want this.
>>>>>
>>>>> On Wed, Aug 19, 2020 at 11:37 AM Holden Karau <holden@pigscanfly.ca>
>>>>> wrote:
>>>>>
>>>>>> Sounds good. In the meantime would folks committing things in core
>>>>>> run the K8s PRB or run it locally? A second change this morning was
>>>>>> committed that broke the K8s PR tests.
>>>>>>
>>>>>> On Tue, Aug 18, 2020 at 9:53 PM Prashant Sharma <scrapcodes@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1, we should enable.
>>>>>>>
>>>>>>> On Wed, Aug 19, 2020 at 9:18 AM Holden Karau <holden@pigscanfly.ca>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Dev Folks,
>>>>>>>>
>>>>>>>> I was wondering how people feel about enabling the K8s PRB
>>>>>>>> automatically for all core changes? Sometimes I forget that
a change might
>>>>>>>> impact one of the K8s integration tests since a bunch of
them look at log
>>>>>>>> messages. Would folks be OK with turning on the K8s integration
PRB for all
>>>>>>>> core changes as well as K8s changes?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Holden :)
>>>>>>>>
>>>>>>>> --
>>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shane Knapp
>>>>> Computer Guy / Voice of Reason
>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>> https://rise.cs.berkeley.edu
>>>>>
>>>>
>>>
>>> --
>>> Shane Knapp
>>> Computer Guy / Voice of Reason
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>


-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Mime
View raw message