metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Richardson <kylerichards...@gmail.com>
Subject Re: [PROPOSAL] up-to-date versioned documentation
Date Fri, 13 Jan 2017 02:19:21 GMT
Matt, thanks for pulling this together. I completely agree that we need to
go all in on either cwiki or the README.md's. I think the wiki is poorly
updated and can cause confusion for new users and devs. My preference is
certainly for the README.md's.

I like your approach but also agree that we shouldn't need to roll our own
here. I really like the Spark documentation that Mike pointed out. Any way
we can duplicate/adapt their approach?

-Kyle

On Thu, Jan 12, 2017 at 7:19 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Casey, Matt - These guys are using doxia
> https://github.com/apache/falcon/tree/master/docs
>
> Honestly, I kind of like Spark's approach -
> https://github.com/apache/spark/tree/master/docs
>
> Mike
>
> On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley <mattf@apache.org> wrote:
>
> > I’m ambivalent; I think we’d end up tied to the doxia processing
> pipeline,
> > which is “yet another arcane toolset” to learn.  Using .md as the input
> > format decreases the dependency, but we’d still be dependent on it.
> >
> > I had anticipated that the web page would be a write-once thing that
> would
> > be only a couple days for an experienced Web developer. But I was going
> to
> > get an estimate from some co-workers before actually trying to get it
> > implemented. And the script is a few hours of work with find and awk.
> >
> > On the other hand, doxia is certainly an expectable solution.  Is setting
> > up that infrastructure less work than developing the web page?  Or is it
> > actually just a matter of a few lines in pom.xml?
> >
> >
> > On 1/12/17, 3:24 PM, "Casey Stella" <cestella@gmail.com> wrote:
> >
> >     Just a followup thought that's a bit more constructive, maybe we
> could
> >     migrate the README.md's into a site directory and use doxia markdown
> >     (example here <https://github.com/larrycai/doxia-markdown-demo>) to
> >     generate the site as part of the build to resolve 1 through 3?
> >
> >     On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella <cestella@gmail.com>
> > wrote:
> >
> >     > So, I do think this would be better than what we currently do.  I
> > like a
> >     > few things in particular:
> >     >
> >     >    - I don't like the wiki one bit.
> >     >    - We have a LOT of documentation in the README.md's and it's
> > sometimes
> >     >    poorly organized
> >     >    - I like a documentation preprocessing pipeline to be present.
> > For
> >     >    instance, a major ask is all of the stellar functions in one
> > place.  That's
> >     >    solved by updating an index manually in the READMEs and keeping
> > it in sync
> >     >    with the annotation.  I'd like to make a stellar annotation ->
> > markdown
> >     >    generator as part of the build and this would be nice for such a
> > task.
> >     >
> >     > My only concern is that the html generation/viewer seems like a
> fair
> >     > amount of engineering.  Are you sure there isn't something easier
> > that we
> >     > could conform to?  I'm sure we aren't the only project in the world
> > that
> >     > has this particular issue.  Is there something like a maven site
> > plugin or
> >     > something?  Just a thought.  I'll come back with more :)
> >     >
> >     > Great ideas!  Keep them coming!
> >     >
> >     > Casey
> >     >
> >     > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <mattf@apache.org>
> > wrote:
> >     >
> >     >> We currently have three forms of documentation, with the following
> >     >> advantages and disadvantages:
> >     >>
> >     >> || Docs || Pro || Con ||
> >     >> | CWiki |
> >     >>       Easy to edit, no special tools required, don't have to be a
> >     >> developer to contribute, google and wiki search |
> >     >> Not versioned, no review process, distant from the code, obsolete
> > content
> >     >> tends to accumulate |
> >     >> | Site |
> >     >>       Versioned and reviewed, only committers can edit, google
> > search |
> >     >>       Yet another arcane toolset must be learned, only web
> > programmers
> >     >> feel comfortable contributing, "asf-site" branch not related to
> code
> >     >> versions, distant from the code, tends to go obsolete due to
> >     >> non-maintenance |
> >     >> | README.md |
> >     >>       Versioned and reviewed, only committers can edit, tied to
> code
> >     >> versions, highly local to the code being documented |
> >     >>       Non-developers don't know about them, may be scared by
> > github, poor
> >     >> scoring in google search, no high-level presentation |
> >     >>
> >     >> Various discussion threads indicate the developer community likes
> >     >> README-based docs, and it's easy to see why from the above.  I
> > propose this
> >     >> extension to the README-based documentation, to address their
> > disadvantages:
> >     >>
> >     >> 1. Produce a script that gathers the README.md files from all code
> >     >> subdirectories into a hierarchical list.  The script would have an
> >     >> exclusion list for non-user-content, which at this point would
> > consist of
> >     >> [site/*, build_utils/*].  The hierarchy would be sorted
> > depth-first.  The
> >     >> resulting hierarchical list at this time (with six added README
> > files to
> >     >> complete the hierarchy) would be:
> >     >>
> >     >> ./README.md
> >     >> ./metron-analytics/README.md  <== (need file here)
> >     >> ./metron-analytics/metron-maas-service/README.md
> >     >> ./metron-analytics/metron-profiler/README.md
> >     >> ./metron-analytics/metron-profiler-client/README.md
> >     >> ./metron-analytics/metron-statistics/README.md
> >     >> ./metron-deployment/README.md
> >     >> ./metron-deployment/amazon-ec2/README.md
> >     >> ./metron-deployment/packaging/README.md  <== (need file here)
> >     >> ./metron-deployment/packaging/ambari/README.md <== (need file
> here)
> >     >> ./metron-deployment/packaging/docker/ansible-docker/README.md
> >     >> ./metron-deployment/packaging/docker/rpm-docker/README.md
> >     >> ./metron-deployment/packer-build/README.md
> >     >> ./metron-deployment/roles/  <== (need file here)
> >     >> ./metron-deployment/roles/kibana/README.md
> >     >> ./metron-deployment/roles/monit/README.md
> >     >> ./metron-deployment/roles/opentaxii/README.md
> >     >> ./metron-deployment/roles/pcap_replay/README.md
> >     >> ./metron-deployment/roles/sensor-test-mode/README.md
> >     >> ./metron-deployment/vagrant/README.md  <== (need file here)
> >     >> ./metron-deployment/vagrant/codelab-platform/README.md
> >     >> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> >     >> ./metron-deployment/vagrant/full-dev-platform/README.md
> >     >> ./metron-deployment/vagrant/quick-dev-platform/README.md
> >     >> ./metron-platform/README.md
> >     >> ./metron-platform/metron-api/README.md
> >     >> ./metron-platform/metron-common/README.md
> >     >> ./metron-platform/metron-data-management/README.md
> >     >> ./metron-platform/metron-enrichment/README.md
> >     >> ./metron-platform/metron-indexing/README.md
> >     >> ./metron-platform/metron-management/README.md
> >     >> ./metron-platform/metron-parsers/README.md
> >     >> ./metron-platform/metron-pcap-backend/README.md
> >     >> ./metron-sensors/README.md  <== (need file here)
> >     >> ./metron-sensors/fastcapa/README.md
> >     >> ./metron-sensors/pycapa/README.md
> >     >>
> >     >> 2. Arrange to run this script as part of the build process, and
> > commit
> >     >> the resulting hierarchy list to someplace in the versioned and
> > branched
> >     >> ./site/ subdirectory.
> >     >>
> >     >> 3. Produce a "doc reader" web page that takes in this hierarchy of
> > .md
> >     >> pages, and presents a LHS doc tree of links, and a main display
> > area for a
> >     >> currently selected file.  If we want to get fancy, this page would
> > also
> >     >> provide: (a) telescoping (collapse/expand) of the doc tree; (b)
> > floating
> >     >> next/prev/up/home buttons in the display area.
> >     >>
> >     >> #4. Add to this web page a pull-down menu that selects among all
> the
> >     >> release versions of Metron, and (if not running in the Apache
> site)
> > a
> >     >> SNAPSHOT version for the current filesystem version (for developer
> >     >> preview).  Let it re-write the file paths per release version to
> > the proper
> >     >> release tag in github.  This web page will therefore be
> >     >> version-independent.  Put it in the asf-site branch of the Apache
> > site, as
> >     >> the new "docs" sub-site from the home web page.  Update the list
> of
> >     >> releases at each release, or if we want to get fancy, teach it to
> > read the
> >     >> release tags from github.
> >     >>
> >     >> 5. As part of the release process, the release manager (a) assures
> > the
> >     >> release is tagged in github with a consistent naming convention,
> > and (b)
> >     >> submits the new hierarchy of links to google search (there's an
> api
> > for
> >     >> that).
> >     >>
> >     >> 6. Deprecate the use of cwiki for anything but long-lived
> >     >> demonstrations/tutorials that are unlikely to go obsolete.
> >     >>
> >     >>
> >     >> Do folks feel this would be a good contribution to the visibility,
> >     >> timeliness, and usability of our docs?
> >     >> Is this an adequate solution for the current problems?
> >     >>
> >     >> Thanks,
> >     >> --Matt
> >     >>
> >     >>
> >     >>
> >     >
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message