airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Travis integration tests and the KubernetesExecutor
Date Fri, 15 Sep 2017 07:27:34 GMT
I can see the k8s integration moving to core at a certain point in time or being maintained
outside of it but as a "core package". Why don't you make the ci build ENV var dependent and
make sure Travis executes it as a separate build or builds?

Bolke

Verstuurd vanaf mijn iPad

> Op 14 sep. 2017 om 22:21 heeft "grantnicholas2015@u.northwestern.edu"<grantnicholas2015@u.northwestern.edu>
het volgende geschreven:
> 
> Hey everyone. I'm working with others on the KubernetesExecutor. As we were building
out pieces of the executor, we realized it would be nice to have an integration test environment
to test the bits of code involving the Kubernetes integration (specifically around crash safety,
but also useful for regression purposes). 
> 
> I did some testing and found running minikube on travis is possible with minikube's `driver=None`
mode, which deploys kubernetes locally instead of in a VM (see here: https://github.com/kubernetes/minikube#linux-continous-integration-with-vm-support).
I spun up a minikube cluster on travis this way and it's definitely technically viable. 
> 
> The issue now is it looks like no other contrib executors have integration tests, so
I'm not sure what pattern I should follow on integration testing (or even if airflow wants
to accept integration tests for this). There would be relatively few tests but they would
be heavyweight (ie: spin up minikube, build airflow + docker image, deploy airflow+postgres
on kubernetes, run tests that spin up dags and then kill the scheduler and assert everything
succeeds after enough time passes for recovery).
> 
> I'm open to ideas, just wanting to get a feel from the community before starting too
much work. 
> 
> 

Mime
View raw message