metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [PROPOSAL] up-to-date versioned documentation
Date Thu, 19 Jan 2017 09:18:44 GMT
Here’s a screen shot, attached :-)

On 1/19/17, 1:04 AM, "Matt Foley" <mattf@apache.org> wrote:

    Hi all,
    I’ve put together a prototype doc book, along the lines we discussed, and it looks pretty
worthwhile.
    Many thanks to Mike M. who whipped the pom.xml file into shape, and helped me find the
right site.xml file to imitate.
    
    If you’re interested, please do a single-branch clone as follows:
        git clone --single-branch -b METRON-660 https://github.com/mattf-horton/incubator-metron.git
 [clonename]
    (or whatever git command pleases you :-)
    
    In this branch, there’s a new top-level subdirectory named site-book/.  This is not
necessarily how we want to integrate stuff, it was just convenient to do it separate from
the existing site/ directory for now.  To build the book, do these three commands:
        cd site-book/
        bin/generate-md.sh   #This gathers all the *.md files into site-book/src/site/markdown/**,
and generates the menu tree into site-book/src/site/site.xml
        mvn clean site:site     #This builds the book with the maven site plugin and doxia-markdown
plugin
    
    If both those steps are successful, you can then go to a browser and open
        file:///Users/yourname/yourpath/clonename/site-book/target/site/index.html
    and see the book, with the nav menu on the LHS.
    It’s important to note that a very usable (not perfect) nav hierarchy has been auto-generated;
this is not hardwired nor hand-edited.
    I do plan to add some overrides that allow individual items in the menu to be tweaked.
    
    While it already looks fairly nice, and clearly illustrates the value of building a book,
there are two glaring issues.
    • Doxia-markdown doesn’t process the triple back-tick (```) the same way as Github
Markdown.  It seems to color-code it as <code>, but doesn’t preserve line breaks,
which is really bad.
    • Similarly, it only processes bullet lists in isolation, and it doesn’t correctly
combine bullet lists subordinate to a numbered list.
    
    The upshot is that 
    • both code and bullet lists often lose their linebreaks, and get mushed into run-on
paragraphs, usually combined with the preceding paragraph, and
    • bullet lists interrupt numbered lists and make them start over at #1.
    
    Perhaps 80-90% of these issues can be fixed by editing the markdown files to put blank
lines around the list formats.  I started doing this, but I didn’t want to obscure the proto
by editing tons of .md files.  As of this proto, only the half dozen actually broken files
(that caused maven site build errors) have been fixed.
    The other 10-20% will just require simplification of the markdown used, unless we can
get an updated version of the plugins.
    
    Anyway, please take a look and share your thoughts.
    
    Thanks,
    --Matt
    
    
    On 1/16/17, 1:02 PM, "Michael Miklavcic" <michael.miklavcic@gmail.com> wrote:
    
        Hey Matt, feel free to ping me.
        
        On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley <mattf@apache.org> wrote:
        
        > I looked into the Falcon website and doxia over the weekend, and I’m
        > convinced that using the doxia-markdown plugin should make it dirt simple
        > to do what’s been discussed in this thread, with no overhead on the part of
        > people writing the README.md files.
        >
        > I fiddled with trying to do a POC, and unfortunately concluded (again)
        > that I don’t really know maven very well :-)
        > Are there any maven experts out there who would be willing to give me some
        > pointers (offline) on how to make use of this apparently simple maven
        > plug-in?
        >
        > I can do the bit of scripting needed to gather the docs.  I’ve opened
        > https://issues.apache.org/jira/browse/METRON-660 with some sub-tasks for
        > this work.
        > --Matt
        >
        > On 1/13/17, 12:04 PM, "Zeolla@GMail.com" <zeolla@gmail.com> wrote:
        >
        >     +1 on any improvement to documentation and more consistency.  At this
        >     point, I think getting rid of or hiding some of the pages on the wiki
        > (at
        >     least for the short term) would be better than leaving them around
        > because
        >     there's a lot of misinformation.
        >
        >     Jon
        >
        >     On Fri, Jan 13, 2017 at 10:13 AM Nick Allen <nick@nickallen.org>
        > wrote:
        >
        >     > +1 I think it is sorely needed.
        >     >
        >     > If we can come up with a really slick solution like Spark, then
        > great. I am
        >     > also not against a half-baked solution that can later evolve into
        > something
        >     > else.  For example, create an index README.md that links together
        > all the
        >     > existing READMEs and run Pandoc on it.  Not ideal, but way better
        > than what
        >     > we have.
        >     >
        >     >
        >     >
        >     > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler <
        > ottobackwards@gmail.com>
        >     > wrote:
        >     >
        >     > > I think something that does what you have laid out here, no matter
        > the
        >     > > implementation details would be ideal
        >     > >
        >     > >
        >     > > On January 12, 2017 at 18:05:24, Matt Foley (mattf@apache.org)
        > wrote:
        >     > >
        >     > > We currently have three forms of documentation, with the following
        >     > > advantages and disadvantages:
        >     > >
        >     > > || Docs || Pro || Con ||
        >     > > | CWiki |
        >     > > Easy to edit, no special tools required, don't have to be a
        > developer to
        >     > > contribute, google and wiki search |
        >     > > Not versioned, no review process, distant from the code, obsolete
        > content
        >     > > tends to accumulate |
        >     > > | Site |
        >     > > Versioned and reviewed, only committers can edit, google search
|
        >     > > Yet another arcane toolset must be learned, only web programmers
        > feel
        >     > > comfortable contributing, "asf-site" branch not related to code
        > versions,
        >     > > distant from the code, tends to go obsolete due to non-maintenance
        > |
        >     > > | README.md |
        >     > > Versioned and reviewed, only committers can edit, tied to code
        > versions,
        >     > > highly local to the code being documented |
        >     > > Non-developers don't know about them, may be scared by github,
poor
        >     > scoring
        >     > > in google search, no high-level presentation |
        >     > >
        >     > > Various discussion threads indicate the developer community likes
        >     > > README-based docs, and it's easy to see why from the above. I
        > propose
        >     > this
        >     > > extension to the README-based documentation, to address their
        >     > > disadvantages:
        >     > >
        >     > > 1. Produce a script that gathers the README.md files from all code
        >     > > subdirectories into a hierarchical list. The script would have
an
        >     > exclusion
        >     > > list for non-user-content, which at this point would consist of
        > [site/*,
        >     > > build_utils/*]. The hierarchy would be sorted depth-first. The
        > resulting
        >     > > hierarchical list at this time (with six added README files to
        > complete
        >     > the
        >     > > hierarchy) would be:
        >     > >
        >     > > ./README.md
        >     > > ./metron-analytics/README.md <== (need file here)
        >     > > ./metron-analytics/metron-maas-service/README.md
        >     > > ./metron-analytics/metron-profiler/README.md
        >     > > ./metron-analytics/metron-profiler-client/README.md
        >     > > ./metron-analytics/metron-statistics/README.md
        >     > > ./metron-deployment/README.md
        >     > > ./metron-deployment/amazon-ec2/README.md
        >     > > ./metron-deployment/packaging/README.md <== (need file here)
        >     > > ./metron-deployment/packaging/ambari/README.md <== (need file
        > here)
        >     > > ./metron-deployment/packaging/docker/ansible-docker/README.md
        >     > > ./metron-deployment/packaging/docker/rpm-docker/README.md
        >     > > ./metron-deployment/packer-build/README.md
        >     > > ./metron-deployment/roles/ <== (need file here)
        >     > > ./metron-deployment/roles/kibana/README.md
        >     > > ./metron-deployment/roles/monit/README.md
        >     > > ./metron-deployment/roles/opentaxii/README.md
        >     > > ./metron-deployment/roles/pcap_replay/README.md
        >     > > ./metron-deployment/roles/sensor-test-mode/README.md
        >     > > ./metron-deployment/vagrant/README.md <== (need file here)
        >     > > ./metron-deployment/vagrant/codelab-platform/README.md
        >     > > ./metron-deployment/vagrant/fastcapa-test-platform/README.md
        >     > > ./metron-deployment/vagrant/full-dev-platform/README.md
        >     > > ./metron-deployment/vagrant/quick-dev-platform/README.md
        >     > > ./metron-platform/README.md
        >     > > ./metron-platform/metron-api/README.md
        >     > > ./metron-platform/metron-common/README.md
        >     > > ./metron-platform/metron-data-management/README.md
        >     > > ./metron-platform/metron-enrichment/README.md
        >     > > ./metron-platform/metron-indexing/README.md
        >     > > ./metron-platform/metron-management/README.md
        >     > > ./metron-platform/metron-parsers/README.md
        >     > > ./metron-platform/metron-pcap-backend/README.md
        >     > > ./metron-sensors/README.md <== (need file here)
        >     > > ./metron-sensors/fastcapa/README.md
        >     > > ./metron-sensors/pycapa/README.md
        >     > >
        >     > > 2. Arrange to run this script as part of the build process, and
        > commit
        >     > the
        >     > > resulting hierarchy list to someplace in the versioned and branched
        >     > ./site/
        >     > > subdirectory.
        >     > >
        >     > > 3. Produce a "doc reader" web page that takes in this hierarchy
of
        > .md
        >     > > pages, and presents a LHS doc tree of links, and a main display
        > area for
        >     > a
        >     > > currently selected file. If we want to get fancy, this page would
        > also
        >     > > provide: (a) telescoping (collapse/expand) of the doc tree; (b)
        > floating
        >     > > next/prev/up/home buttons in the display area.
        >     > >
        >     > > #4. Add to this web page a pull-down menu that selects among all
        > the
        >     > > release versions of Metron, and (if not running in the Apache
        > site) a
        >     > > SNAPSHOT version for the current filesystem version (for developer
        >     > > preview). Let it re-write the file paths per release version to
the
        >     > proper
        >     > > release tag in github. This web page will therefore be
        >     > version-independent.
        >     > > Put it in the asf-site branch of the Apache site, as the new "docs"
        >     > > sub-site from the home web page. Update the list of releases at
        > each
        >     > > release, or if we want to get fancy, teach it to read the release
        > tags
        >     > from
        >     > > github.
        >     > >
        >     > > 5. As part of the release process, the release manager (a) assures
        > the
        >     > > release is tagged in github with a consistent naming convention,
        > and (b)
        >     > > submits the new hierarchy of links to google search (there's an
        > api for
        >     > > that).
        >     > >
        >     > > 6. Deprecate the use of cwiki for anything but long-lived
        >     > > demonstrations/tutorials that are unlikely to go obsolete.
        >     > >
        >     > >
        >     > > Do folks feel this would be a good contribution to the visibility,
        >     > > timeliness, and usability of our docs?
        >     > > Is this an adequate solution for the current problems?
        >     > >
        >     > > Thanks,
        >     > > --Matt
        >     > >
        >     >
        >     >
        >     >
        >     > --
        >     > Nick Allen <nick@nickallen.org>
        >     >
        >     --
        >
        >     Jon
        >
        >     Sent from my mobile device
        >
        >
        >
        >
        >
        
    
    
    
    
    


Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message