nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Processor documentation plugin
Date Sun, 08 Feb 2015 13:21:40 GMT
I like it! Nice work, Dan. I appreciate all the work you’ve put into this. I know it’s
not been trivial by any means!


Unless others object, I will pull this in tomorrow (Monday) morning.






From: Daniel Bress
Sent: ‎Sunday‎, ‎February‎ ‎8‎, ‎2015 ‎4‎:‎32‎ ‎AM
To: dev@nifi.incubator.apache.org





All,
   I tweaked the layout of the generated documentation and put a new rendered copy here: http://danbress.github.io/generated-documentation/components-02072015/

   Please provide any feedback you may have.  I changed the css for the table a little bit
to make things more readable.  I'm not married to it, so if anyone wants to build upon it,
please go forward.

   I am cancelling my old pull request and issuing a new one as I did the changes for this
in a new branch.  Let me know how this looks.

Dan Bress
Software Engineer
ONYX Consulting Services

________________________________________
From: Daniel Bress <dbress@onyxconsults.com>
Sent: Sunday, January 25, 2015 9:43 AM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin

Jenn,
   Good point, your text makes a lot more sense than what I had.  Thanks for that!  I just
included it and regenerated the docs, let me know what you think.

Dan Bress
Software Engineer
ONYX Consulting Services

________________________________________
From: Jennifer Barnabee <jennifer.barnabee@gmail.com>
Sent: Sunday, January 25, 2015 9:15 AM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin

Dan,
It looks good. Assuming we decide to move forward with it, we'll want to
change the boilerplate paragraph under Properties to suit what you have a
little better. I propose something like the following, but it may need to
change as things evolve. Please see what you think...

In the list below, the names of required properties appear in *bold*. Any
other properties (not in bold) are considered optional. The table also
indicates any default values, whether a property supports the NiFi
Expression Language (or simply EL), and whether a property is considered
"sensitive", meaning that its value will be encrypted. Before entering a
value in a sensitive property, ensure that the *nifi.properties* file has
an entry for the property

*nifi.sensitive.props.key.*
-Jenn

On Sun, Jan 25, 2015 at 9:01 AM, Daniel Bress <dbress@onyxconsults.com>
wrote:

> Tony,
>    Thanks for pointing that out.  I had a "\" when I needed a "/".  I
> fixed this and re uploaded it.  Let me know how that looks.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Tony Kurc <trkurc@gmail.com>
> Sent: Saturday, January 24, 2015 8:07 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Dan,
> For whatever reason, when I follow the hyperlinks, the last "slash" is a \
> instead of a / and I get a 404 (
>
> http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html
> ).
>
>
>
>
> On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <dbress@onyxconsults.com>
> wrote:
>
> > All,
> >     I've generated the first round of documentation I'd like everyone to
> > look at to see if they like this new format.  Its really only slightly
> > changed from the current format(using tables instead of lists for
> > Properties and Relationships), but its all machine generated, except for
> > the stuff linked by "Additional Details".
> >
> >    Again, this would be generated by the app at startup.  I'd also like
> to
> > have this generated sometime during the build and linked from the
> website.
> > For now I just pulled that generated html and uploaded it to my github
> page
> > for your review.
> >
> > http://danbress.github.io/generated-documentation/components/
> >
> > Let me know what you think!  I'm open to any ideas/suggestions.
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <markap14@hotmail.com>
> > Sent: Thursday, January 22, 2015 8:16 AM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > OK, now I understand where you're coming from. Sorry, I misunderstood the
> > idea before. If you think it would be too verbose then I think we should
> > hold off for now. I think it's best to keep it simple. If we want to add
> it
> > in later, we can. If we add it now and decide it's too verbose, it would
> be
> > awkward to go back and remove documentation later but it's easy to later
> > add in something that's missing.
> >
> > So in short I agree with your assessment :)
> > -Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Thu, 22 Jan 2015 03:13:16 +0000
> > >
> > > Mark,
> > >    No worries.  On the dynamic property tip, I see what you are saying.
> > I was hoping to provide the same level of detail in describing dynamic
> > properties as non-dynamic ones, but right now I see no way to do that.
> > Thats fine, I'll put that part(providing detailed documentation of the
> > dynamic properties) on the back burner.
> > >
> > >    On the Validator tip, for each PropertyDescriptor I am getting the
> > list of AllowableValues so that I can say "This property has X, Y and Z
> for
> > valid values".  If there are no AllowableValues present I would like to
> > look at the validators and say "This property requires a positive
> > integer".  or "This requires a <data size> followed by a <data unit>".
> > Although typing this all out now makes it seem like it might be too
> verbose
> > and cluttered.  I think I'll forget this idea and just work with
> > AllowableValues.
> > >
> > > Thanks again,
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <markap14@hotmail.com>
> > > Sent: Tuesday, January 20, 2015 12:16 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > Sorry, I never did respond to this e-mail.
> > > You cannot get the dynamic PropertyDescriptor because the
> > AbstractConfigurableComponent (the base abstract class from which every
> > processor, controller service, reporting task that I know of extends)
> > provides the following method signature:
> > > protected PropertyDescriptor
> getSupportedDynamicPropertyDescriptor(final
> > String propertyDescriptorName)
> > > I.e., there is not a PropertyDescriptor for dynamic properties but
> > rather the descriptor can be different for every dynamic property. For
> > example, consider a Processor that allows dynamic properties but for
> every
> > property X there must be 2 other properties: X.format (any string) and
> > X.size (a non-zero integer). If there were a single PropertyDescriptor,
> > this wouldn't be possible because all dynamic properties would have the
> > same validators, etc. But by making a property descriptor based on the
> > property name, we can easily achieve this.
> > > Regarding the second note: I don't think Validator can provide
> > information about what it is validating because it doesn't know. For
> > instance, PositiveInteger validator just validates that some value is a
> > positive integer. It does not (and I think should not) know the context
> of
> > the value.
> > > If anything here doesn't make sense or if you think it could be done
> > better, let me know and we can discuss further.
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Fri, 16 Jan 2015 21:14:41 +0000
> > > >
> > > > Mark & Matt,
> > > >    Cool.  While I like the idea of @SupportsDynamicProperties,
> > especially if it has a description, I'd also like a guaranteed way to get
> > the dynamic PropertyDescriptor so I can interrogate it further.  I'd be
> > interested in knowing whether it supports expression language, and/or how
> > the value of the property is validated.
> > > >
> > > > On that note, I was thinking it may be helpful to add either an
> > annotation or a method to the Validator interface that could provide a
> text
> > description about what it is validating.  Is this useful or overkill?
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <markap14@hotmail.com>
> > > > Sent: Friday, January 16, 2015 11:14 AM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > It might make sense to have an annotation for this:
> > > > @SupportsDynamicProperties
> > > > Perhaps even allow developer to provide a string for how they are
> used:
> > > > @SupportsDynamicProperties("Name of dynamic property becomes
> attribute
> > name; value becomes attribute value.")
> > > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6
> > but I have not pushed anything to a branch or anything yet. Basically, I
> > deprecated the existing annotations and created 3 new packages for them.
> > Instead of org.apache.nifi.processor.annotation, we will have:
> > > >
> >
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > > > So the @SupportsDynamicProperties could go under 'behavior' if that
> > makes sense.
> > > >
> > > > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > > > Subject: Re: Processor documentation plugin
> > > > > From: matt.c.gilman@gmail.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > >
> > > > > Dan,
> > > > >
> > > > > We discussed item 1 awhile back and even created a ticket for it.
> > However,
> > > > > there was always higher priority work that took precedence. I am
> not
> > sure
> > > > > if that ticket was ever created in JIRA for Apache NiFi. If not,
we
> > should.
> > > > > I believe the current mechanism allows the processor to decide if
a
> > given
> > > > > dynamic property should be a supported. I think we would need a
> more
> > > > > general indicator that dynamic properties are supported at all to
> > trigger
> > > > > that button state. This would likely help with item 2 as well.
> > > > >
> > > > > Matt
> > > > >
> > > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > wrote:
> > > > >
> > > > > > Mark,
> > > > > >    Sounds good.  I'm on board.  I was thinking of having the
> > > > > > ControllerServices/ReportingTask's documentation include an
XML
> > template,
> > > > > > but if those will be configurable via the UI then that will
not
> be
> > needed.
> > > > > >    I'm on board with your rev 1 / rev 2 suggestions too.  I'll
> get
> > to work
> > > > > > accordingly.
> > > > > >
> > > > > >
> > > > > > Matt/Mark,
> > > > > > Two UI/API questions:
> > > > > > 1) If a processor does not support dynamic properties, why does
> > the UI let
> > > > > > them hit [+] New Property?
> > > > > > 2) If I have an instance of Processor class, I can call
> > > > > > getSupportedProperties() on it to get a list of the
> > required/optional
> > > > > > properties.  But I don't have a way to know if that processor
> > supports
> > > > > > dynamic properties or not.  Supporting dynamic properties or
not
> > was
> > > > > > something I was planning on reflecting in the documentation.
 Is
> > there any
> > > > > > way to detect this currently?
> > > > > >
> > > > > > Thanks,
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <markap14@hotmail.com>
> > > > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: RE: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > > I think this is great!
> > > > > > I would recommend we slate this for the 0.1.0 version, which
> > includes
> > > > > > putting Controller Services and Reporting Tasks in the UI. But
by
> > the same
> > > > > > token I think that makes your first bulletin for "rev 2" more
of
> a
> > priority
> > > > > > for "rev 1" if we can get it there. I could pitch in and help
if
> > needed.
> > > > > > I'd also recommend you put the Capability Description in for
Rev
> 1
> > as
> > > > > > that's arguably the most important part of the docs (which may
> > make the
> > > > > > Tags very easy at that point??)
> > > > > > The rest of what you have listed for "rev 2" i would agree should
> > be there.
> > > > > >
> > > > > > Very excited to see this happening!
> > > > > > Thanks-Mark
> > > > > >
> > > > > > > From: dbress@onyxconsults.com
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > > > >
> > > > > > > Mark et al,
> > > > > > >    Ok, I think I hear everyone's concerns and ideas.  I
will
> > challenge
> > > > > > myself to come up with a solution that makes everyone happy
all
> > the time.
> > > > > > >
> > > > > > >    The goals of this effort will be:
> > > > > > >       - Machine generate as much of the basic(processor
> > description,
> > > > > > property description, relationship description) info as possible.
> > This
> > > > > > will make the developers lives better by storing this information
> > in one
> > > > > > place, and make users lives better by having the "inline"
> > documentation
> > > > > > match the "usage" documentation.  This will also ensure that
all
> > the
> > > > > > documentation is formatted consistently.
> > > > > > >       - Allow the developer to optionally provide additional
> > html and
> > > > > > images that would be linked from the machine generated
> > documentation
> > > > > > >       - Provide a way for the documentation to be generated
> > statically
> > > > > > and linked from the Apache NiFi webpage
> > > > > > >
> > > > > > >     A few other 'nice-to-have' goals(probably for rev 2):
> > > > > > >       - Support documenting ControllerServices and
> > ReportingTasks is a
> > > > > > similar way
> > > > > > >           -Maybe leverage existing annotations on these
types?
> > E.g
> > > > > > @Tags, @CapabilityDescription...
> > > > > > >       - Provide new 'documentation' annotations that would
> allow
> > a
> > > > > > developer to describe the FlowFile attributes a processor uses
on
> > input,
> > > > > > and which attributes a processor modifies on output
> > > > > > >       - Provide new 'documentation' annotations that describe
> the
> > > > > > expected input format of the flow file content
> > > > > > >       - Provide new 'documentation' annotations that describe
> > whether
> > > > > > the processor modifies the flow file content
> > > > > > >       - Provide new 'documentation' annotations that describe
> > the format
> > > > > > of the outbound flow file content
> > > > > > >       - Provide new 'documentation' annotations that describe
> > other NiFi
> > > > > > artifacts(processors, controller services, reporting tasks)
that
> > are
> > > > > > related to this one, e.g. "See Also ..."
> > > > > > >       - Consider if/how current Processor annotations should
be
> > > > > > displayed (@SideEffectFree, @TriggerSerially,
> @TriggerWhenEmpty...,
> > > > > > @EventDriven)
> > > > > > >
> > > > > > >
> > > > > > >     High level implementation plan:
> > > > > > >        - Develop a utility class to read an
> AbstractProcessor(and
> > > > > > existing HTML documention) and generated the basic documentation
> > with links
> > > > > > to existing HTML
> > > > > > >        - Integrate this class into the framework such that
all
> > > > > > documentation displayed within a NiFi instance is generated
by
> the
> > utility
> > > > > > >        - Integrate this utility into something that can
take a
> > NiFi
> > > > > > build, dig through the nars, find the
> > > > > > Processors/ControllerServices/ReportingTasks and generate their
> > appropriate
> > > > > > HTML documentation so that it can be stored somewhere and linked
> > from the
> > > > > > Apache NiFi website.
> > > > > > >
> > > > > > >     If this feels like the wrong path, or if you have any
other
> > ideas
> > > > > > please speak up.  Thanks!
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Mark Payne <markap14@hotmail.com>
> > > > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: RE: Processor documentation plugin
> > > > > > >
> > > > > > > Dan,
> > > > > > > Yes, that's what I'm saying. Right now, it's all done manually
> > by hand,
> > > > > > and it's a pain. The down side i think to doing it via a Maven
> > plugin is
> > > > > > that if the user wants to modify it, they have to rebuild, pull
> the
> > > > > > generated html out of the target directory, and then modify
it
> and
> > put it
> > > > > > in the src/main/resources/docs. Then, if they change something
> > they have to
> > > > > > do this again. If it's done on-the-fly, rather than building
> > static content
> > > > > > with a plugin, that doesn't have to happen.
> > > > > > > But you may come up with something much better than I'm
> > envisioning :)
> > > > > > Having a util to do it is a good call though. Then it could
be
> > used in a
> > > > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > > > Thanks-Mark
> > > > > > >
> > > > > > > > From: dbress@onyxconsults.com
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > > > >
> > > > > > > > Mark,
> > > > > > > >    Sorry.. I missed your reply in the thread.
> > > > > > > >
> > > > > > > >    Are you saying that going forward, you want to
auto
> > generate the
> > > > > > doc and allow the developer to include a "more info" link? 
I
> > wanted to
> > > > > > confirm that it doesn't work that way now.
> > > > > > > >
> > > > > > > >    I'll think about this a little more.  Might write
some
> code
> > to
> > > > > > generate the documentation from an AbstractProcessor class,
then
> > decide
> > > > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Dan Bress
> > > > > > > > Software Engineer
> > > > > > > > ONYX Consulting Services
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Mark Payne <markap14@hotmail.com>
> > > > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > >
> > > > > > > > Dan,
> > > > > > > >
> > > > > > > > Matt and I discussed this approach before but ultimately
> > decided that
> > > > > > we like the idea of just auto generating the docs on demand
with
> a
> > "More
> > > > > > Info" type of link that lets you add more in html format if
you
> > care to do
> > > > > > so.
> > > > > > > >
> > > > > > > > Others' thoughts and ideas are welcome though!
> > > > > > > >
> > > > > > > > Sent from my iPhone
> > > > > > > >
> > > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > >   Just pulled down the code out of Git and got
everything
> up
> > and
> > > > > > running on Windows.  This was pretty simple, and the UI looks
> > great under
> > > > > > chrome.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >   I was thinking of writing a maven plugin to
generate the
> > processor
> > > > > > usage HTML documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >   Looking through the existing processor documentation,
it
> > seems
> > > > > > like most of this information comes directly from the code in
the
> > > > > > processors, via @CapabilityDescription, and the
> > PropertyDescriptors and
> > > > > > Relationships.  Then if your processor only requires this level
> of
> > > > > > documentation in the HTML format, you could tie this plugin
to
> the
> > build,
> > > > > > and keep your inline processor documentation in sync with the
> HTML
> > > > > > documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >   What do you think?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Dan Bress
> > > > > > > > > Software Engineer
> > > > > > > > > ONYX Consulting Services
> > > > > >
> >
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message