airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <>
Subject Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client
Date Tue, 11 Aug 2020 19:44:52 GMT
-1 from me without a firm plan how we will replace it.

I see keeping it and extending to use the new API would ensure that everything the CLI can
do locally (i.e. when airflow webserver isn't up yet, with the ) also works over the API with
the exception of db utilities.


On 11 August 2020 20:05:56 BST, QP Hou <> wrote:
>+1 for replacing the existing remote mode client with the open api
>client. IMO, we don't really have other options here because the
>experimental API will be deprecated in the future.
>For OpenAPI based Airflow REST clients, the current plan is to maintain
>the code gen automation within the main source tree [1], then use it to
>populate each individual language specific client repo like the go
>mentioned earlier. So far, we have the go client completed and
>validated to
>make sure this development flow will meet our needs. The next step I
>the community should focus on is getting API auth implemented [2]
>before we
>move on to generate the python client. How we do API auth could have a
>impact on client code gen automation, so it is worth waiting for.
>Once we have authentication implemented in both Airflow core and
>we should be all good to start doing version releases for our API
>That said, adopting open api based clients in the CLI alone won't
>the issue of CLI depending on full airflow installation. Some of the
>commands like `dags test` depend on a full airflow installation by
>We will have to either develop a separate CLI intended for remote only
>or add a flag in the existing cli so it can run in a pure remote mode
>it would disable loading of code that requires airflow installation
>On Tue, Aug 11, 2020 at 11:05 AM Kamil BreguĊ‚a
>> Hello,
>> I think we should remove remote mode in CLI and in-core API Client
>> (airflow.api.client package).
>> Here is docs about remote mode:
>> Since these features were introduced, it has never been actively
>> and I don't think it's widely used. At the same time, Apache Airflow
>> evolving, and this code stands out more and more from the rest.
>> My main reservations about these features:
>> - Remote mode/in-core API Client is rarely used. I asked a few people
>> none of them used it in production. Does anyone use it?
>> - A very small number of commands are available (7 pools command and
>2 dags
>> command only)
>> - Remote mode/API Client depends on experimental REST API.
>> - Remote mode/API Client is a handwritten code that is difficult to
>> maintain.
>> - No documentation for API client
>> - Remote mode/API Client has low test coverage.
>> - Remote mode does not provide a good level of security, because it
>> on experimental API. There is the only authentication, but the
>> authenticated user can perform any operation.
>> - Requires full Airflow to be installed along with a large number of
>> unnecessary dependencies. Some of them are difficult to install in
>> environments, e.g. setproctitle on Windows
>> - Using this client API changes the logger configuration because it
>> requires importing the airflow package.
>> I think this remote mode in CLI is something valuable, but I think we
>> do it in a different way in the future, e.g. generate a CLI/API
>> based on the OpenAPI specification.
>> Generated API clients can be installed independently of airflow and
>will be
>> easier to maintain. We already have one API client for golang
>> in this way, so new languages will only be developing this idea.
>> -
>> I will be happy to discuss the vision of the development of these two
>> things. How do we want to develop these two things?
>> Best regards,
>> Kamil Bregula

  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message