airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client
Date Tue, 18 Aug 2020 11:01:04 GMT
Fine for me. Deprecating might be deferred, I have no problem with that :).

On Thu, Aug 13, 2020 at 11:08 PM Kaxil Naik <kaxilnaik@gmail.com> wrote:

> I feel deprecating without having a replacement will not aid anything.
>
> "deprecations" are marked just to give users a warning and I feel we have
> enough warnings for the user in the API endpoint itself where we say it is
> "experimental".
>
> https://airflow.readthedocs.io/en/stable/rest-api-ref.html says "The API
> structure is not stable. We expect the endpoint definitions to change.
>
> I feel we should keep the endpoints until 2.1.0 until users migrate to new
> endpoints or just remove them in 2.0, we already have a table that map old
> endpoints to new endpoints:
> https://github.com/apache/airflow/blob/master/UPDATING.md#base-endpoint
>
> Regards,
> Kaxil
>
> "
>
> On Thu, Aug 13, 2020 at 8:55 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> wrote:
>
> > Yep I agree removal (especially now in 1.10) is not a good idea.That's
> fine
> > not to remove it in 2.0 but disable by default.
> >
> > But I still think we should mark the whole API deprecated NOW (in
> 1.10.12).
> >
> > It would be great to have at least one release in 1.10. where we
> explicitly
> > mark it as deprecated. We already know that we are going to use the new
> API
> > in 2.0 and it's pretty much "ready" to be released - so I'd say - let's
> > mark it as deprecated even if we do not provide a direct replacement
> > immediately, just to let know people to whe will try to use it now, that
> > they should think twice before doing it.  I don't think "@deprecated"
> means
> > "we provide a direct replacement already". It's more " do not use it
> > because it will disappear". Whether it is replaced by something else or
> not
> > is an entirely different story.
> >
> > J.
> >
> >
> > On Thu, Aug 13, 2020 at 9:18 PM Ry Walker <ry@rywalker.com> wrote:
> >
> > > Yes I agree the new API would be made default, and experimental would
> be
> > > disabled by default.
> > >
> > > Before the new API goes live, we should allow minor improvements to the
> > > Experimental API (for example
> > https://github.com/apache/airflow/pull/10308
> > > ,
> > > which addresses to some degree the security hole in the experimental
> API
> > > for users of Airflow RBAC) which could change your last sentence a
> > little,
> > > Kamil.
> > >
> > >
> > > On Thu, Aug 13, 2020 at 1:54 PM Kamil Breguła <
> kamil.bregula@polidea.com
> > >
> > > wrote:
> > >
> > > > I agree. We do not have to completely delete the experimental API in
> > > > Airflow 2.0, but I think it is worth turning off by default so that
> the
> > > > user has to make a conscious decision that they want to use the API,
> > > which
> > > > provides a limited level of security-  no permission control, all
> > > > authorized users have full power.
> > > >
> > > > On Thu, Aug 13, 2020 at 7:42 PM QP Hou <qph@scribd.com> wrote:
> > > >
> > > > > Yeah, i am also curious to know more about the reason why we want
> to
> > > nuke
> > > > > the experimental api code soon instead of just marking it as
> > > deprecated.
> > > > >
> > > > > As for getting more insights into remote mode cli usage, would it
> > make
> > > > > sense to make it part of the airflow user survey?
> > > > >
> > > > > Thanks,
> > > > > QP Hou
> > > > >
> > > > >
> > > > > On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <ry@rywalker.com>
wrote:
> > > > >
> > > > > > I would think we would deprecate the old API once we say the
new
> > API
> > > is
> > > > > > “ready to go” - and leave it in place a while as users transition
> > to
> > > > new
> > > > > > API. Why is there an urgency to remove it from codebase?
> > > > > >
> > > > > > On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <
> ash@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Removing the experimental is a fundamental breaking change
to
> > > users'
> > > > > > > workflows, and so we should remove it before 2.0.
> > > > > > >
> > > > > > > -ash
> > > > > > >
> > > > > > > On 13 August 2020 10:14:02 BST, Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com
> > > > > >
> > > > > > > wrote:
> > > > > > > >And I think we should make the whole experimenta API
> deprecated
> > in
> > > > > > > >1.10.12
> > > > > > > >possibly ?
> > > > > > > >
> > > > > > > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> > > > > > > ><Jarek.Potiuk@polidea.com>
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> I think no matter what, Maybe we should simply
make it
> > > deprecated
> > > > in
> > > > > > > >the
> > > > > > > >> upcoming (today?) release of 1.10.12 ? Then we
can decide if
> > in
> > > -
> > > > > > > >> potential  - 1.10.13 we remove it or leave it
as it is.
> > > > > > > >>
> > > > > > > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> > > > > > > ><kamil.bregula@polidea.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> I started this thread mainly to discuss what
we want to do
> > with
> > > > > this
> > > > > > > >>> remote
> > > > > > > >>> mode prior to the Airflow 2.0 release. This
is mainly due
> to
> > > the
> > > > > > > >fact that
> > > > > > > >>> he is using an experimental API which will
be deprecated.
> > > > > > > >>>
> > > > > > > >>> In my opinion, we have several solutions.
> > > > > > > >>> a) Delete this mode as unused and not supported.
> > > > > > > >>> b) Rewrite in-core API client to support stable
API
> > > > > > > >>> c) Prepare OpenAPI based client and rewrite
CLI to use it
> > > > > > > >>> d) Leave as-is
> > > > > > > >>>
> > > > > > > >>> There were various expectations on the mailing
list about
> > this
> > > > > mode,
> > > > > > > >but I
> > > > > > > >>> haven't seen anyone actively contribute to
it. This brings
> me
> > > > > > > >further
> > > > > > > >>> questions. Does anyone use this mode in its
current form?
> If
> > no
> > > > one
> > > > > > > >is
> > > > > > > >>> using it, I think we can take more radical
steps to start
> > with
> > > a
> > > > > > > >blank
> > > > > > > >>> page. This will make it easier to start work
and be able to
> > > > iterate
> > > > > > > >over
> > > > > > > >>> it
> > > > > > > >>> faster in the future. This looks like a simple
task, but if
> > we
> > > > want
> > > > > > > >to be
> > > > > > > >>> sure that no breaking-changes are made, we
should pay off
> > some
> > > > > > > >technical
> > > > > > > >>> debt and increase testing coverage before
we can think
> about
> > > > making
> > > > > > > >more
> > > > > > > >>> changes. It may not be necessary if we choose
a different
> > path
> > > of
> > > > > > > >>> development.
> > > > > > > >>>
> > > > > > > >>> I think now is a good time to try to make
some decisions if
> > > > someone
> > > > > > > >is
> > > > > > > >>> actually interested in developing these features.
I do not
> > > think
> > > > we
> > > > > > > >need a
> > > > > > > >>> precise vision of the development, if currently
this
> feature
> > is
> > > > not
> > > > > > > >used
> > > > > > > >>> by
> > > > > > > >>> anyone and no one is really interested in
its development.
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qph@scribd.com>
> > wrote:
> > > > > > > >>>
> > > > > > > >>> > I think it's best to divide the discussion
into two
> > separate
> > > > > > > >topics.
> > > > > > > >>> >
> > > > > > > >>> > First one is to replace the existing
json_client with the
> > new
> > > > to
> > > > > > > >be
> > > > > > > >>> created
> > > > > > > >>> > official Airflow Python Client backed
by the new RESTful
> > API.
> > > > > This
> > > > > > > >IMHO
> > > > > > > >>> is
> > > > > > > >>> > a must have considering we are to deprecate
experimental
> > API
> > > > > going
> > > > > > > >>> forward.
> > > > > > > >>> >
> > > > > > > >>> > The second topic is to create a better
CLI experience
> > > > leveraging
> > > > > > > >the new
> > > > > > > >>> > APIs. This is much more controversial.
I remember us
> > having a
> > > > > > > >>> > similar discussion in the dev list a
year ago, which
> didn't
> > > get
> > > > > > > >much
> > > > > > > >>> > traction. It's not possible to fit all
existing CLI
> > > > > > > >functionalities into
> > > > > > > >>> > REST APIs. DB utils use-case that Ash
mentioned is a
> great
> > > > > > > >example. So I
> > > > > > > >>> > think one potential solution is to split
the CLI into
> two.
> > > Keep
> > > > > > > >the
> > > > > > > >>> > existing CLI as the admin/management
CLI can communicate
> > > > directly
> > > > > > > >with
> > > > > > > >>> the
> > > > > > > >>> > DB and taps into airflow core code base.
On the other
> hand,
> > > we
> > > > > can
> > > > > > > >>> create a
> > > > > > > >>> > separate user facing CLI that's light
weight, fast and
> > remote
> > > > > > > >only. It
> > > > > > > >>> > doesn't even need to be written in python
to make it
> easier
> > > to
> > > > > > > >>> distribute
> > > > > > > >>> > as a single binary.
> > > > > > > >>> >
> > > > > > > >>> > Thanks,
> > > > > > > >>> > QP Hou
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash
Berlin-Taylor
> > > > > > > ><ash@apache.org>
> > > > > > > >>> wrote:
> > > > > > > >>> >
> > > > > > > >>> > > -1 from me without a firm plan how
we will replace it.
> > > > > > > >>> > >
> > > > > > > >>> > > I see keeping it and extending to
use the new API would
> > > > ensure
> > > > > > > >that
> > > > > > > >>> > > everything the CLI can do locally
(i.e. when airflow
> > > > webserver
> > > > > > > >isn't
> > > > > > > >>> up
> > > > > > > >>> > > yet, with the ) also works over
the API with the
> > exception
> > > of
> > > > > db
> > > > > > > >>> > utilities.
> > > > > > > >>> > >
> > > > > > > >>> > > -ash
> > > > > > > >>> > >
> > > > > > > >>> > > On 11 August 2020 20:05:56 BST,
QP Hou <qph@scribd.com
> >
> > > > wrote:
> > > > > > > >>> > > >+1 for replacing the existing
remote mode client with
> > the
> > > > open
> > > > > > > >api
> > > > > > > >>> > > >based
> > > > > > > >>> > > >client. IMO, we don't really
have other options here
> > > because
> > > > > > > >the
> > > > > > > >>> > > >experimental API will be deprecated
in the future.
> > > > > > > >>> > > >
> > > > > > > >>> > > >For OpenAPI based Airflow REST
clients, the current
> plan
> > > is
> > > > to
> > > > > > > >>> maintain
> > > > > > > >>> > > >all
> > > > > > > >>> > > >the code gen automation within
the main source tree
> [1],
> > > > then
> > > > > > > >use it
> > > > > > > >>> to
> > > > > > > >>> > > >populate each individual language
specific client repo
> > > like
> > > > > the
> > > > > > > >go
> > > > > > > >>> > > >client
> > > > > > > >>> > > >mentioned earlier. So far, we
have the go client
> > completed
> > > > and
> > > > > > > >>> > > >validated to
> > > > > > > >>> > > >make sure this development flow
will meet our needs.
> The
> > > > next
> > > > > > > >step I
> > > > > > > >>> > > >think
> > > > > > > >>> > > >the community should focus on
is getting API auth
> > > > implemented
> > > > > > > >[2]
> > > > > > > >>> > > >before we
> > > > > > > >>> > > >move on to generate the python
client. How we do API
> > auth
> > > > > could
> > > > > > > >have
> > > > > > > >>> a
> > > > > > > >>> > > >big
> > > > > > > >>> > > >impact on client code gen automation,
so it is worth
> > > waiting
> > > > > > > >for.
> > > > > > > >>> > > >
> > > > > > > >>> > > >Once we have authentication
implemented in both
> Airflow
> > > core
> > > > > > > >and
> > > > > > > >>> > > >clients,
> > > > > > > >>> > > >we should be all good to start
doing version releases
> > for
> > > > our
> > > > > > > >API
> > > > > > > >>> > > >clients.
> > > > > > > >>> > > >
> > > > > > > >>> > > >That said, adopting open api
based clients in the CLI
> > > alone
> > > > > > > >won't
> > > > > > > >>> > > >address
> > > > > > > >>> > > >the issue of CLI depending on
full airflow
> installation.
> > > > Some
> > > > > > > >of the
> > > > > > > >>> > > >cli
> > > > > > > >>> > > >commands like `dags test` depend
on a full airflow
> > > > > installation
> > > > > > > >by
> > > > > > > >>> > > >design.
> > > > > > > >>> > > >We will have to either develop
a separate CLI intended
> > for
> > > > > > > >remote
> > > > > > > >>> only
> > > > > > > >>> > > >use
> > > > > > > >>> > > >or add a flag in the existing
cli so it can run in a
> > pure
> > > > > > > >remote mode
> > > > > > > >>> > > >where
> > > > > > > >>> > > >it would disable loading of
code that requires airflow
> > > > > > > >installation
> > > > > > > >>> > > >entirely.
> > > > > > > >>> > > >
> > > > > > > >>> > > >[1]:
> > > https://github.com/apache/airflow/tree/master/clients,
> > > > > > > >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> > > > > > > >>> > > >
> > > > > > > >>> > > >On Tue, Aug 11, 2020 at 11:05
AM Kamil Breguła
> > > > > > > >>> > > ><kamil.bregula@polidea.com>
> > > > > > > >>> > > >wrote:
> > > > > > > >>> > > >
> > > > > > > >>> > > >> Hello,
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> I think we should remove
remote mode in CLI and
> > in-core
> > > > API
> > > > > > > >Client
> > > > > > > >>> > > >> (airflow.api.client package).
> > > > > > > >>> > > >> Here is docs about remote
mode:
> > > > > > > >>> > > >>
> > > > > > > >>> > > >>
> > > > > > > >>> > > >
> > > > > > > >>> > >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> Since these features were
introduced, it has never
> > been
> > > > > > > >actively
> > > > > > > >>> > > >developed
> > > > > > > >>> > > >> and I don't think it's
widely used. At the same
> time,
> > > > Apache
> > > > > > > >>> Airflow
> > > > > > > >>> > > >is
> > > > > > > >>> > > >> evolving, and this code
stands out more and more
> from
> > > the
> > > > > > > >rest.
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> My main reservations about
these features:
> > > > > > > >>> > > >> - Remote mode/in-core API
Client is rarely used. I
> > > asked a
> > > > > > > >few
> > > > > > > >>> people
> > > > > > > >>> > > >and
> > > > > > > >>> > > >> none of them used it in
production. Does anyone use
> > it?
> > > > > > > >>> > > >> - A very small number of
commands are available (7
> > pools
> > > > > > > >command
> > > > > > > >>> and
> > > > > > > >>> > > >2 dags
> > > > > > > >>> > > >> command only)
> > > > > > > >>> > > >> - Remote mode/API Client
depends on experimental
> REST
> > > API.
> > > > > > > >>> > > >> - Remote mode/API Client
is a handwritten code that
> is
> > > > > > > >difficult to
> > > > > > > >>> > > >> maintain.
> > > > > > > >>> > > >> - No documentation for
API client
> > > > > > > >>> > > >> - Remote mode/API Client
has low test coverage.
> > > > > > > >>> > > >> - Remote mode does not
provide a good level of
> > security,
> > > > > > > >because it
> > > > > > > >>> > > >depends
> > > > > > > >>> > > >> on experimental API. There
is the only
> authentication,
> > > but
> > > > > > > >the
> > > > > > > >>> > > >> authenticated user can
perform any operation.
> > > > > > > >>> > > >> - Requires full Airflow
to be installed along with a
> > > large
> > > > > > > >number
> > > > > > > >>> of
> > > > > > > >>> > > >> unnecessary dependencies.
Some of them are difficult
> > to
> > > > > > > >install in
> > > > > > > >>> > > >some
> > > > > > > >>> > > >> environments, e.g. setproctitle
on Windows
> > > > > > > >>> > > >> - Using this client API
changes the logger
> > configuration
> > > > > > > >because it
> > > > > > > >>> > > >> requires importing the
airflow package.
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> I think this remote mode
in CLI is something
> valuable,
> > > > but I
> > > > > > > >think
> > > > > > > >>> we
> > > > > > > >>> > > >can
> > > > > > > >>> > > >> do it in a different way
in the future, e.g.
> generate
> > a
> > > > > > > >CLI/API
> > > > > > > >>> > > >Client
> > > > > > > >>> > > >> based on the OpenAPI specification.
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> Generated API clients can
be installed independently
> > of
> > > > > > > >airflow and
> > > > > > > >>> > > >will be
> > > > > > > >>> > > >> easier to maintain. We
already have one API client
> for
> > > > > golang
> > > > > > > >>> > > >implemented
> > > > > > > >>> > > >> in this way, so new languages
will only be
> developing
> > > this
> > > > > > > >idea.
> > > > > > > >>> > > >> - https://github.com/apache/airflow-client-go
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> I will be happy to discuss
the vision of the
> > development
> > > > of
> > > > > > > >these
> > > > > > > >>> two
> > > > > > > >>> > > >> things. How do we want
to develop these two things?
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> Best regards,
> > > > > > > >>> > > >> Kamil Bregula
> > > > > > > >>> > > >>
> > > > > > > >>> > >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >>
> > > > > > > >> Jarek Potiuk
> > > > > > > >> Polidea <https://www.polidea.com/> | Principal
Software
> > > Engineer
> > > > > > > >>
> > > > > > > >> M: +48 660 796 129 <+48660796129>
> > > > > > > >> [image: Polidea] <https://www.polidea.com/>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > > >--
> > > > > > > >
> > > > > > > >Jarek Potiuk
> > > > > > > >Polidea <https://www.polidea.com/> | Principal
Software
> > Engineer
> > > > > > > >
> > > > > > > >M: +48 660 796 129 <+48660796129>
> > > > > > > >[image: Polidea] <https://www.polidea.com/>
> > > > > > >
> > > > > > --
> > > > > > Sent from Gmail Mobile
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message