airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client
Date Tue, 11 Aug 2020 19:44:52 GMT
-1 from me without a firm plan how we will replace it.

I see keeping it and extending to use the new API would ensure that everything the CLI can
do locally (i.e. when airflow webserver isn't up yet, with the ) also works over the API with
the exception of db utilities.

-ash

On 11 August 2020 20:05:56 BST, QP Hou <qph@scribd.com> wrote:
>+1 for replacing the existing remote mode client with the open api
>based
>client. IMO, we don't really have other options here because the
>experimental API will be deprecated in the future.
>
>For OpenAPI based Airflow REST clients, the current plan is to maintain
>all
>the code gen automation within the main source tree [1], then use it to
>populate each individual language specific client repo like the go
>client
>mentioned earlier. So far, we have the go client completed and
>validated to
>make sure this development flow will meet our needs. The next step I
>think
>the community should focus on is getting API auth implemented [2]
>before we
>move on to generate the python client. How we do API auth could have a
>big
>impact on client code gen automation, so it is worth waiting for.
>
>Once we have authentication implemented in both Airflow core and
>clients,
>we should be all good to start doing version releases for our API
>clients.
>
>That said, adopting open api based clients in the CLI alone won't
>address
>the issue of CLI depending on full airflow installation. Some of the
>cli
>commands like `dags test` depend on a full airflow installation by
>design.
>We will have to either develop a separate CLI intended for remote only
>use
>or add a flag in the existing cli so it can run in a pure remote mode
>where
>it would disable loading of code that requires airflow installation
>entirely.
>
>[1]: https://github.com/apache/airflow/tree/master/clients,
>[2]: https://github.com/apache/airflow/issues/8112
>
>On Tue, Aug 11, 2020 at 11:05 AM Kamil BreguĊ‚a
><kamil.bregula@polidea.com>
>wrote:
>
>> Hello,
>>
>> I think we should remove remote mode in CLI and in-core API Client
>> (airflow.api.client package).
>> Here is docs about remote mode:
>>
>>
>https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
>>
>> Since these features were introduced, it has never been actively
>developed
>> and I don't think it's widely used. At the same time, Apache Airflow
>is
>> evolving, and this code stands out more and more from the rest.
>>
>> My main reservations about these features:
>> - Remote mode/in-core API Client is rarely used. I asked a few people
>and
>> none of them used it in production. Does anyone use it?
>> - A very small number of commands are available (7 pools command and
>2 dags
>> command only)
>> - Remote mode/API Client depends on experimental REST API.
>> - Remote mode/API Client is a handwritten code that is difficult to
>> maintain.
>> - No documentation for API client
>> - Remote mode/API Client has low test coverage.
>> - Remote mode does not provide a good level of security, because it
>depends
>> on experimental API. There is the only authentication, but the
>> authenticated user can perform any operation.
>> - Requires full Airflow to be installed along with a large number of
>> unnecessary dependencies. Some of them are difficult to install in
>some
>> environments, e.g. setproctitle on Windows
>> - Using this client API changes the logger configuration because it
>> requires importing the airflow package.
>>
>> I think this remote mode in CLI is something valuable, but I think we
>can
>> do it in a different way in the future, e.g. generate a CLI/API
>Client
>> based on the OpenAPI specification.
>>
>> Generated API clients can be installed independently of airflow and
>will be
>> easier to maintain. We already have one API client for golang
>implemented
>> in this way, so new languages will only be developing this idea.
>> - https://github.com/apache/airflow-client-go
>>
>> I will be happy to discuss the vision of the development of these two
>> things. How do we want to develop these two things?
>>
>> Best regards,
>> Kamil Bregula
>>

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message