airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: [DISCUSS] Moving stuff from CWiki to Github ?
Date Mon, 03 Aug 2020 14:57:38 GMT
Tomek, I would love to leave aside ADR snapshots for now - we can have a
separate discussion about it later on how to do it.

I think we need to clean up the mess we have now in CWIKI. I would love to
focus only on this for now.

I think we all agree we should only keep planning/overview of the effort.
However, currently, it's NOT the case for Airflow 2.0 and Cwiki (we have
way too much overlapping information):

1) https://cwiki.apache.org/confluence/display/AIRFLOW/Roadmap - general
outdated roadmap
2) https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 -
High-level features (Pretty much the same as in the
https://github.com/apache/airflow/issues/10085)
3)
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning -
this is I think a bit different - recently updated by Kaxil and it is
mostly about general approach we should take (and overlap with the rest is
minimal).

My proposal:

3) should stay and we should archive/deprecate 1) and 2).

I also have the feeling that we must archive all the stuff that is
outdated because it is really confusing now which information is outdated
or not.

My proposal:

* AIPs - we keep them for now as they are (And make it part of later ADR
discussion).
* Airflow Links - archive & deprecate
* Airflow Release Planning - we leave it for now and update it as part of
2.0 discussion
* Building Docs - archive & deprecate
* Releasing Airflow - I can move it to "dev" together with planned backport
release doc updates
* Announcements - leave it for now (maybe forever)
* API conventions - archive & deprecate
* Committers/Commiter's Guide -> archive & deprecate  (review if some
information can be move to CONTRIBUTING.rst)
* Common Pitfalls -> move it to docs
* Community Gudelines, contributor's Guide -> archive & deprecate  (review
if some information can be move to CONTRIBUTING.rst)
* First time contributor's workshop -> move it as blog to apache.airflow.org
* File lists - > move it to Airflow repo as resources.
* Meeting notes -> archive & deprecate.
* Meetups -> archive & deprecate.
* Product requirements, Roadmap Airflow 2.0 -> archive & deprecate
* Roles -> archive & deprecate (some stuff moved to CONTRIBUTING.rst)
* Scheduler Basics - > move it to docs
* Season of Docs 2019 -> archive & deprecate

Does it sound reasonable? Does anyone think some other things should stay?
I am happy to do it if no-one objects.

J.


On Mon, Aug 3, 2020 at 3:42 PM Tomasz Urbaszek <turbaszek@apache.org> wrote:

> I agree with Ry. Moreover,we should adjust cwiki information after a
> discussion on devlist/github. So I think in the case of AIPs the cwiki
> should work as someting like architecture decision records.
> However, I'm afraid that there will be no way to automate or enforce
> this synchronization.
>
> Tomek
>
>
> On Mon, Aug 3, 2020 at 3:32 PM Ry Walker <ry@rywalker.com> wrote:
> >
> > I'd say the cwiki should provide an overview of the effort, as it does
> now,
> > and that we should keep track of the work in a Github project using
> github
> > issues. The cwiki should link to that project board as the source of
> truth
> > for project status. This will help the wiki page to be perceived as up to
> > date as it won't need to be updated with each bit of progress.
> >
> > -Ry
> >
> >
> > On Mon, Aug 3, 2020 at 4:29 AM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> > wrote:
> >
> > > Yeah. After experimenting a bit with it - seems that Wiki in Github is
> > > a bit "abandoned" place and omissions like lack of auto-linking issues
> > > and PRs is a big bummer.
> > >
> > > Kamil - would you mind re-creating the issue based on the old issue? I
> > > - unfortunately - added all "Apache Committers" to it so we cannot
> > > re-open it.
> > >
> > > But I have another question here:
> > >
> > > 1) Should we remove
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
> > > completely?
> > > 2) More than that - should we archive and move everything else from
> > > CWiki to Github Issues?
> > >
> > >
> > > I think it will be very confusing (especially for new contributors) if
> > > we keep some information in CWiki but also start using Github Issues
> > > for similar purpose. So I would be for archiving all content in the
> > > CWiki and moving it all to Issues.
> > >
> > > I took a look at the kind of documents we have in CWiki and we have a
> > > LOT of information there that is outdated or could live elsewhere.
> > > Here are my proposals:
> > >
> > > * Airflow 2.0 planing - we could completely move it to "Airflow 2.0
> > > Release" issue
> > > * AIPs - we could keep all the completed AIP-s there (And keep the
> > > "proposed" ones for the future) but we could move all the "active"
> > > AIPs to Github Issues and add all the new AIPs there.
> > > * Airflow Links - we can abandon it (It's already abandoned in fact -
> > > last update May 2017)
> > > * Airflow Release Planning - we could review it and turn it into a
> > > "meta" issue - it has a lot fo information about pre-1.10 releases
> > > which we can remove (And we will have to redefine it after we agree
> > > release schedule and versioning for 2.* series)
> > > * Building Docs - is outdated
> > > * Releasing Airflow - I think we can move it to Airflow's source code
> > > in "dev" folder (like I did for the Backport Packages)
> > > * Announcements -> that one we might do on "airflow.apache.org" site
> > > as a Blog post ?
> > > * API conventions - outdated
> > > * Committers/Commiter's Guide -> we could have it in the
> > > "CONTRIBUTING.rst" documentation of Airflow (some of the information
> > > there is not valid anyway and CONTRIBUTING documentation is much more
> > > updated)
> > > * Common Pitfalls -> I think that one belongs to the documentation of
> > > Airflow not to Wiki and we could select/move some still valid
> > > information from there to the documentation
> > > * Community Gudelines, contributor's Guide -> this all in
> CONTRIBUTING.rst
> > > * First time contributor's workshop -> this can be moved to a
> > > "apache.airflow.org" as a Blog Post.
> > > * File lists - > those files can be all added to the airflow
> > > repository in "resources" folder or smth.
> > > * Meeting notes - Those could be  added to relevant issues in GitHub.
> > > We could have "meta" issues for "special interest groups" and add
> > > meeting notes there.
> > > * Meetups -> already part of airflow.apache.org
> > > * Product requirements, Roadmap Airflow 2.0 -> this all could be moved
> > > to "meta" issues
> > > * Roles -> should be added to CONTRIBUTING.rst
> > > * Scheduler Basics - > should be part of Airflow Documentation
> > >  * Season of Docs 2019 -> we can archive it.
> > >
> > > We could also use Github Wiki to only have "Index" of all important
> > > issues that are "permanent" - Airflow 2.0 roadmap, Special interest
> > > groups, AIPs,
> > >
> > > Let me know what you think?
> > >
> > > J.
> > >
> > >
> > >
> > > On Sun, Aug 2, 2020 at 12:31 PM Kaxil Naik <kaxilnaik@gmail.com>
> wrote:
> > > >
> > > > I agree with Tomek and feel Github issues ("meta"-issue) is a better
> > > place
> > > > than Github Wiki.
> > > >
> > > > On Sun, Aug 2, 2020 at 11:26 AM Tomasz Urbaszek <
> turbaszek@apache.org>
> > > > wrote:
> > > >
> > > > > I see the advantage of having no comment in wiki but in the longer
> > > > > run, I think this will create confusion. Where should I discuss a
> > > > > particular thing? On devlist? Slack? In issue? How should a new
> > > > > contributor know this?
> > > > >
> > > > > After giving some thought to that I'm leaning towards the
> meta-issue:
> > > > > - they are clear (no need to go to wiki)
> > > > > - give possibilit to link other issues/PRs that shows their
> content on
> > > > > hover
> > > > > - this is great advantage as we can see how our work is
> interconnected
> > > > > - having an issue make it explicit to where contributors should
> leave
> > > > > their comments
> > > > >
> > > > > No matter what we decide, we should thrive to limit the places
> where
> > > > > information is available.
> > > > >
> > > > > Bests,
> > > > > Tomek
> > > > >
> > > > >
> > > > > On Sun, Aug 2, 2020 at 12:00 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Question. Should we move over Airflow 2.0 Status and other
> > > "permanent"
> > > > > > information to Github Wiki? See here for example:
> > > > > > https://github.com/apache/airflow/wiki/Airflow-2.0
> > > > > >
> > > > > > The discussion originated by Kamil creating an issue for Airflow
> 2.0
> > > -
> > > > > > which was essentially overriding the page we had in
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
> > > > > > and adding more "status" information in
> > > > > > https://github.com/apache/airflow/issues/10085. This was more
> of a
> > > > > > "meta" issue as it has a lot of unrelated issues / projects
> mentioned
> > > > > > - the only common thing for those was that it was "Airflow 2.0".
> But
> > > > > > we already have "Milestone 2.0" and CWIKI page.
> > > > > >
> > > > > > My proposal was that since we have 2.0 Milestone already we
> should
> > > use
> > > > > > this one to mark issues for 2.0 and in order to keep
> > > > > > Roadmap/Plans/Status we can use Github's Wiki instead. IMHO
it is
> > > much
> > > > > > better as it does not allow comments - which is good IMHO. For
> this
> > > > > > jind of "permanent" pages, comments and discussion should happen
> for
> > > > > > the individual issues not for the page itself  (especially when
> you
> > > do
> > > > > > not have in-line comments).
> > > > > >
> > > > > > And this page should always be "current" - with the old roadmap
> in
> > > > > > CWIKI and the issue 10085 when you add comments, you quickly
lose
> > > > > > track whether the comments are more important than the overview,
> and
> > > > > > how accurate the "overview" is.  When you just edit the wiki
-
> you
> > > > > > always do it deliberately - because you want to update status
> rather
> > > > > > than make a comment or discuss,
> > > > > >
> > > > > > So I created this as copy of the issue:
> > > > > > https://github.com/apache/airflow/wiki/Airflow-2.0 so that we
> can
> > > > > > compare it - can you please compare it with
> > > > > > https://github.com/apache/airflow/issues/10085 and voice your
> > > opinion
> > > > > > what's better?
> > > > > >
> > > > > > I think it's also a great opportunity to archive a lot of the
> old and
> > > > > > not up-to-date from the old Wiki and migrate it to GitHub. We
> could
> > > > > > move AIPs to Github issues (as needed) - AIPS are fine for
> > > > > > discussion/issues/comments, but when they got implemented we
> could
> > > > > > move it over to wiki as "Implemented" status for history.
> > > > > >
> > > > > > Let me know what you think.
> > > > > >
> > > > > > BTW. PLEASE do NOT comment on that #10085 issue (it's now locked
> and
> > > > > > closed). I accidentally (shame on me) notified all Apache
> Committers.
> > > > > > Happened twice today (also for someone else) so I opened a
> ticket to
> > > > > > Infra to restrict that (If only possible) because it's all too
> easy
> > > to
> > > > > > notify everyone @Apache). If you comment there 3K+ people get
> > > > > > notified.
> > > > > >
> > > > > > But feel free to upvote the infra ticket:
> > > > > > https://issues.apache.org/jira/browse/INFRA-20623
> > > > > >
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Jarek Potiuk
> > > > > > Polidea | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > >
> > > M: +48 660 796 129
> > >
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message