airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Nicholas <grantnicholas2...@u.northwestern.edu>
Subject Re: Travis integration tests and the KubernetesExecutor
Date Fri, 15 Sep 2017 18:11:55 GMT
Awesome. I'll try setting it up as a separate build in the Travis matrix, I
agree that seems like a good place to put it.

On Sep 15, 2017 2:27 AM, "Bolke de Bruin" <bdbruin@gmail.com> wrote:

> I can see the k8s integration moving to core at a certain point in time or
> being maintained outside of it but as a "core package". Why don't you make
> the ci build ENV var dependent and make sure Travis executes it as a
> separate build or builds?
>
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 14 sep. 2017 om 22:21 heeft "grantnicholas2015@u.northwestern.edu"<
> grantnicholas2015@u.northwestern.edu> het volgende geschreven:
> >
> > Hey everyone. I'm working with others on the KubernetesExecutor. As we
> were building out pieces of the executor, we realized it would be nice to
> have an integration test environment to test the bits of code involving the
> Kubernetes integration (specifically around crash safety, but also useful
> for regression purposes).
> >
> > I did some testing and found running minikube on travis is possible with
> minikube's `driver=None` mode, which deploys kubernetes locally instead of
> in a VM (see here: https://github.com/kubernetes/minikube#linux-continous-
> integration-with-vm-support). I spun up a minikube cluster on travis this
> way and it's definitely technically viable.
> >
> > The issue now is it looks like no other contrib executors have
> integration tests, so I'm not sure what pattern I should follow on
> integration testing (or even if airflow wants to accept integration tests
> for this). There would be relatively few tests but they would be
> heavyweight (ie: spin up minikube, build airflow + docker image, deploy
> airflow+postgres on kubernetes, run tests that spin up dags and then kill
> the scheduler and assert everything succeeds after enough time passes for
> recovery).
> >
> > I'm open to ideas, just wanting to get a feel from the community before
> starting too much work.
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message