spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shane knapp ☠ <skn...@berkeley.edu>
Subject Re: minikube and kubernetes cluster versions for integration testing
Date Wed, 03 Mar 2021 17:49:53 GMT
please open a jira for this and assign it to me...  shouldn't be too big of
a deal to get this set up.

On Tue, Mar 2, 2021 at 6:06 PM Dongjoon Hyun <dongjoon.hyun@gmail.com>
wrote:

> Thank you for sharing and suggestion, Attila.
>
> Additionally, given the following information,
>
> - The latest Minikube is v1.18.0 with K8s v1.20.2
> - AWS EKS will add K8s v1.20 on April, 2021
> - The end of support in AWS EKS are
>     K8s v1.15 (May 3, 2021)
>     K8s v1.16 (July, 2021)
>     K8s v1.17 (September, 2021)
>
> The minimum K8s versions (v1.17) sound reasonable and practical to me for
> Apache Spark 3.2.0.
>
> For Minikube, I'd like to recommend to use the latest Minikube versions.
> However, if Minikube v1.7.3 support is easy enough in the script, +1 for
> using v1.7.3 as the minimum Minikube version checking.
>
> Thanks,
> Dongjoon.
>
>
> On Tue, Mar 2, 2021 at 5:03 AM Attila Zsolt Piros <
> piros.attila.zsolt@gmail.com> wrote:
>
>> Hi All,
>>
>> I am working on PR to change kubernetes integration testing and use the
>> `minikube kubectl -- config view --minify` output to build the kubernetes
>> client config.
>> This solution has the advantage of not using hardcoded values like 8443
>> for server port (which is wrong when the vm-driver is docker as the port in
>> that case is 32788 by default).
>>
>> But my question is bit more generic than my PR. It is about the supported
>> Minikube versions and kubernetes cluster version this why I decided to
>> write this mail.
>>
>> To test this new solution I have created shell script to install each
>> Minikube versions one by one, start a kubernetes cluster and view the
>> config with the command above.
>> Running the test I found some issues.
>>
>> Currently for k8s testing we suggest to use *minikube version v0.34.1 or
>> greater* with *kubernetes version v1.15.12* (for details check "Testing
>> K8S" section in the developer tools page
>> <https://spark.apache.org/developer-tools.html>).
>>
>>
>> *The following three findings I have:*
>> 1) Looking the Minikube documentation I came across an advice
>> <https://minikube.sigs.k8s.io/docs/handbook/config/#selecting-a-kubernetes-version>
>> about checking which kubernetes cluster versions are supported for a
>> Minikube version:
>>
>>
>> *"For up to date information on supported versions,
>> see OldestKubernetesVersion and NewestKubernetesVersion in constants.go
>> <https://github.com/kubernetes/minikube/blob/master/pkg/minikube/constants/constants.go>"*
>> I think it would be a good idea to follow the official support matrix
>> of Minikube so I have collected some relevant versions into this table (the
>> link navigates to the relevant lines in `constants.go`):
>>                      |           kubernetes version               |
>> minikube version     |    oldest    |  newest      | default      |
>> ====================================================================
>> v0.34.1
>> <https://github.com/kubernetes/minikube/blob/v0.34.1/pkg/minikube/constants/constants.go#L133>
>>              |    ???       |    ???       | v1.13.3      |
>> v1.1.0 (22 May 2019)
>> <https://github.com/kubernetes/minikube/blob/v1.1.0/pkg/minikube/constants/constants.go#L164-L171>
>> | v1.10.13     | v1.14.2      | v1.14.2      |
>> v1.2.0
>> <https://github.com/kubernetes/minikube/blob/v1.2.0/pkg/minikube/constants/constants.go#L164-L171>
>>               | v1.10.13     | v1.15.0      | v1.15.0      |
>> v1.3.0 (6 Aug 2019)
>> <https://github.com/kubernetes/minikube/blob/v1.3.0/pkg/minikube/constants/constants.go#L178-L185>
>>  | v1.10.13     | v1.15.2      | v1.15.2      |
>> v1.6.0 (11 Dec 2019)
>> <https://github.com/kubernetes/minikube/blob/v1.6.0/pkg/minikube/constants/constants.go#L67-L74>
>> | v1.11.10     | v1.17.0      | v1.17.0      |
>> v1.7.3 (8 Feb 2020)
>> <https://github.com/kubernetes/minikube/blob/436667c819c324e35d7e839f8116b968a2d0a3ff/pkg/minikube/constants/constants.go#L30-L35>
>> | v1.11.10     | v1.17.3      | v1.17.3      |
>> v1.13.1
>> <https://github.com/kubernetes/minikube/blob/v1.13.1/pkg/minikube/constants/constants.go#L29-L35>
>>              | v1.13.0      | v1.19.2      | v1.19.2      |
>> v1.17.1
>> <https://github.com/kubernetes/minikube/blob/v1.17.1/pkg/minikube/constants/constants.go#L30-L36>
>>             | v1.13.0      | v1.20.2      | v1.20.3-rc.0 |
>>
>>
>> Looking this we can see if we intend to support v1.15.12 as kubernetes
>> version we should drop everything under v1.3.0.
>>
>> 2) I would suggest to drop v1.15.12 as kubernetes
>> version version because of this issue
>> <https://github.com/kubernetes/minikube/issues/10663> (I just found it
>> by running my script).
>>
>> 3) On Minikube v1.7.2 there is this permission denied issue
>> <https://github.com/kubernetes/minikube/issues/6583> so I suggest to
>> support Minikube version 1.7.3 and greater.
>>
>> My test script is check_minikube_versions.zsh
>> <https://gist.github.com/attilapiros/8648a782e0b956b59f03f914c88c2df3#file-check_minikube_versions-zsh>.
It
>> was executed on Mac but with a simple sed expression it can be tailored to
>> linux too.
>>
>>
>>
>> *After all of this my questions:*
>> *A) What about to change the required versions and suggest to use
>> kubernetes v1.17.3 and Minikube v1.7.3 and greater for integration testing?*
>>
>> I would chose v1.17.3 for k8s cluster as that is the newest supported k8s
>> version for that Minikube v1.7.3 (hoping it will be good for us for a long
>> time).
>> If you agree with this suggestion I go ahead and update the relevant
>> documentation.
>>
>>
>>
>> *B) How about extending the integration test to check whether the
>> Minikube version is sufficient? *By this we can provide a meaningful
>> error when it is violated.
>>
>> Bests,
>> Attila
>>
>

-- 
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Mime
View raw message