airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "grantnicholas2015@u.northwestern.edu"<grantnicholas2...@u.northwestern.edu>
Subject Travis integration tests and the KubernetesExecutor
Date Thu, 14 Sep 2017 20:21:59 GMT
Hey everyone. I'm working with others on the KubernetesExecutor. As we were building out pieces
of the executor, we realized it would be nice to have an integration test environment to test
the bits of code involving the Kubernetes integration (specifically around crash safety, but
also useful for regression purposes). 

I did some testing and found running minikube on travis is possible with minikube's `driver=None`
mode, which deploys kubernetes locally instead of in a VM (see here: https://github.com/kubernetes/minikube#linux-continous-integration-with-vm-support).
I spun up a minikube cluster on travis this way and it's definitely technically viable. 

The issue now is it looks like no other contrib executors have integration tests, so I'm not
sure what pattern I should follow on integration testing (or even if airflow wants to accept
integration tests for this). There would be relatively few tests but they would be heavyweight
(ie: spin up minikube, build airflow + docker image, deploy airflow+postgres on kubernetes,
run tests that spin up dags and then kill the scheduler and assert everything succeeds after
enough time passes for recovery).

I'm open to ideas, just wanting to get a feel from the community before starting too much
work. 



Mime
View raw message