airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil BreguĊ‚a <kamil.breg...@polidea.com>
Subject [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client
Date Tue, 11 Aug 2020 18:05:19 GMT
Hello,

I think we should remove remote mode in CLI and in-core API Client
(airflow.api.client package).
Here is docs about remote mode:
https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance

Since these features were introduced, it has never been actively developed
and I don't think it's widely used. At the same time, Apache Airflow is
evolving, and this code stands out more and more from the rest.

My main reservations about these features:
- Remote mode/in-core API Client is rarely used. I asked a few people and
none of them used it in production. Does anyone use it?
- A very small number of commands are available (7 pools command and 2 dags
command only)
- Remote mode/API Client depends on experimental REST API.
- Remote mode/API Client is a handwritten code that is difficult to
maintain.
- No documentation for API client
- Remote mode/API Client has low test coverage.
- Remote mode does not provide a good level of security, because it depends
on experimental API. There is the only authentication, but the
authenticated user can perform any operation.
- Requires full Airflow to be installed along with a large number of
unnecessary dependencies. Some of them are difficult to install in some
environments, e.g. setproctitle on Windows
- Using this client API changes the logger configuration because it
requires importing the airflow package.

I think this remote mode in CLI is something valuable, but I think we can
do it in a different way in the future, e.g. generate a CLI/API Client
based on the OpenAPI specification.

Generated API clients can be installed independently of airflow and will be
easier to maintain. We already have one API client for golang implemented
in this way, so new languages will only be developing this idea.
- https://github.com/apache/airflow-client-go

I will be happy to discuss the vision of the development of these two
things. How do we want to develop these two things?

Best regards,
Kamil Bregula

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message