airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sid Anand <san...@apache.org>
Subject Re: Use Docker for running Airflow tests
Date Thu, 06 Sep 2018 01:45:40 GMT
Any chance you can document a flow that ties the CI build artifacts to this
repo and how we would use them? Would our build times go down?

-s

On Sun, Aug 19, 2018 at 11:43 AM Driesprong, Fokko <fokko@driesprong.frl>
wrote:

> Hi all,
>
> Gerardo is doing some awesome work on Dockerizing the CI pipeline. The PR
> is still open here: https://github.com/apache/incubator-airflow/pull/3393
>
> To make this work, and not store any Docker images in private repositories,
> I've created a new repository which stores the Docker image which contains
> a lot of the dependencies we use to test against. The repository is here:
> https://github.com/apache/incubator-airflow-ci
> For most of the tests, we use mocks, but sometimes we also use the actual
> service, for example for some Hadoop, Kerberos, Hive functionality. The
> images are build in the Dockerhub service:
> https://hub.docker.com/r/airflowci/incubator-airflow-ci/builds/
> I've created it for now. I'm open for another name, but Airflow was already
> taken, and hyphens are not allowed.
>
> This change will greatly decouple our dependency on Travis and will make it
> much easier to test the code locally. Instead of setting up tox
> environments, we can just use Docker instead. I'm really enthusiastic about
> the change, and really thankful for Gerardo to put all this effort in
> there.
>
> Recently Holden Karau did a lifestream
> <https://www.youtube.com/watch?v=5Il3w2P7IcM>, and did her first PR. I
> think the most important learning from it was that testing was not as
> trivial as it should be. Therefore I think we should see if we can merge
> the PR from Gerardo soon.
>
> Cheers, Fokko
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message