nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Srinivasan <james.sriniva...@gmail.com>
Subject Re: Lowering the barrier of entry
Date Sat, 26 Jan 2019 13:22:57 GMT
That would have been great - the only things I would add are
reading/writing attributes & reading the processor's properties. Maybe
something about testing in NiFi using GenerateFlowFile too?

I ended up referring to the bundled processors a lot, but it was
sometimes hard to see the wood for the trees.

On Fri, 25 Jan 2019 at 23:04, Andy LoPresto <alopresto@apache.org> wrote:
>
> James,
>
> I’m wondering if a page outlining a toy processor (something like `CountText` or `ReverseContent`)
and doing a line-by-line annotation from a developer’s perspective would be helpful. It
could be a few sections:
>
> 1. How to get to this point
>         * running the maven archetype
>         * choosing the directory to install to
>         * putting the class name in the manifest file
>         * etc.
> 2. The code
>         * here’s the class
>         * here’s what extending AbstractProcessor gets you, etc. A lot of this is currently
in the Developer Guide, but in textbook mode
>         * here’s how to modify some code (i.e. write one line of Java that switches
it from straight content pass-through to reversing the text)
>         * here’s how to make a unit test (introduce the TestRunner framework, etc.)
> 3. Running, building, installing
>         * Run your unit test from the IDE/maven
>         * Build the NAR module
>         * Install the NAR in NiFi lib/ or custom/
>         * Restart NiFi
>                 * See the NAR loaded in the log
>                 * Deploy the component on the canvas
>
> I imagine this being written more conversationally/blog-like than most of our current
reference documentation to be used as a split-screen walkthrough. Each section could certainly
link to the existing detailed documentation for various topics, like the processor lifecycle,
etc.
>
> Does this sounds like something that would have helped you?
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Jan 25, 2019, at 1:59 PM, James Srinivasan <james.srinivasan@gmail.com>
wrote:
> >
> > 9) Oh, and the wiki is a little hard to navigate and the contents rather patchy
> >
> > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > <james.srinivasan@gmail.com> wrote:
> >>
> >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> >> realise I could and possibly should submit PRs :)
> >>
> >> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> >> it would have been nice it if set up unit tests for me.
> >> 2) The User and Developer documentation is great and comprehensive.
> >> Finding the developer docs is a little painful (handful of items at
> >> the end of a scrolling list of 200+ processors)
> >> 3) The Developer docs could possibly do with a little more clarity on
> >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> >> over the docs, it looks pretty clear now
> >> 4) Some example code for common operations e.g. getting/setting
> >> attributes or reading/writing/modifying flowfile content would be
> >> great.
> >> 5) When using existing processors for inspiration, best practices
> >> weren't always clear e.g. some generated properties inside
> >> getSupportedPropertyDescriptors(), others generated a private static
> >> list on init and returned that. Such differences are inevitable in a
> >> large project, but it would be nice to have something blessed to start
> >> from.
> >> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen)
> >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> >> the great blog posts by Matt Burgess and others were invaluable
> >> 8) In case this all sounds too negative, NiFi is fab!
> >>
> >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <aperepel@gmail.com> wrote:
> >>>
> >>> I am not against the archetype. But we need to spell out every step of the
> >>> way. I'd like to see a user thinking about their custom logic ASAP rather
> >>> than fighting the tools to get started. Those steps should be brain-dead,
> >>> just reflexes, if you know what I mean. Hell, let them create a custom
> >>> processor project or prototype in a script by accident even! :)
> >>>
> >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <bbende@gmail.com> wrote:
> >>>
> >>>> That makes sense about the best practice for deploying to an
> >>>> additional lib directory.
> >>>>
> >>>> So for the project structure you are saying it would be easier to have
> >>>> a repo somewhere with essentially the same thing that is in the
> >>>> archetype, but they just clone it and rename it themselves (what the
> >>>> archetype does for you)?
> >>>>
> >>>> Something that I think would be awesome is if we could provide a
> >>>> web-based project initializer that would essentially run the archetype
> >>>> behind the scenes and then let you download the archive of the code,
> >>>> just like the spring-boot starter [1]. Not sure if their initializr
is
> >>>> something that can be re-used and customized [2].
> >>>>
> >>>> The problem is we would need to host that somewhere.
> >>>>
> >>>> [1] https://start.spring.io/
> >>>> [2] https://github.com/spring-io/initializr
> >>>>
> >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <aperepel@gmail.com>
wrote:
> >>>>>
> >>>>> We assume they create new projects from archetypes every day. They
don't.
> >>>>>
> >>>>> We also assume they know how to deploy new NARs. Most don't. Especially
> >>>> if
> >>>>> we want them to follow best practices and create an additional NAR
> >>>> bundles
> >>>>> directory entry im the config (vs dumping into nifi lib).
> >>>>>
> >>>>> I can attest that I feel a bit lost myself every time I need to
come back
> >>>>> to this and refresh my brain synapses. If we could make these not
require
> >>>>> any of that and make simple thinga dead simple....
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <bbende@gmail.com>
wrote:
> >>>>>
> >>>>>> Andrew,
> >>>>>>
> >>>>>> I'm not disagreeing with your points, but I'm curious how you
see
> >>>>>> those two ideas being different from the processor archetype
and the
> >>>>>> wiki page with the archetype commands?
> >>>>>>
> >>>>>> Is it just that people don't know about it?
> >>>>>>
> >>>>>> -Bryan
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> >>>>>>
> >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <ottobackwards@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I think this ties into my other discuss thread on refreshing
the
> >>>>>> archetypes
> >>>>>>>
> >>>>>>>
> >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande (aperepel@gmail.com)
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I consistently see my users struggling when they move up
the nifi
> >>>> food
> >>>>>>> chain and start looking at custom processors. The good content
about
> >>>>>>> prototyping processsors via scripting processors and finalizing
with
> >>>> a
> >>>>>> full
> >>>>>>> NAR bundle is everywhere but where it should be.
> >>>>>>>
> >>>>>>> A few simple changes could help (not *more* docs). They
are great,
> >>>> much
> >>>>>>> better than in many other projecta, but people are already
drowning
> >>>> in
> >>>>>>> those.
> >>>>>>>
> >>>>>>> How about:
> >>>>>>>
> >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op
to fill
> >>>> in
> >>>>>> is
> >>>>>>> miles better than a blank text area (which invokes a blank
stare).
> >>>>>>>
> >>>>>>> + As much as we may loook down on this, but... A simple
guide to a
> >>>> full
> >>>>>> NAR
> >>>>>>> build as a series of copy/paste commands.
> >>>>>>>
> >>>>>>> There's more, but this should fit the context for now.
> >>>>>>>
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <mikerthomsen@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> One of the changes we should make is to create a separate
guide for
> >>>>>>> product
> >>>>>>>> vendors on how to build and maintain a bundle. We're
at that point
> >>>>>> where
> >>>>>>>> vendors will have to do it on their own as extension
providers, so
> >>>> it
> >>>>>>> would
> >>>>>>>> be very helpful for them to have a simple and straight
forward
> >>>> document
> >>>>>>>> showing them what should be there, best practices for
> >>>> maintainability
> >>>>>> and
> >>>>>>>> where to announce it.
> >>>>>>>>
> >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <bbende@gmail.com>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> I think we have a lot more documentation than most
projects, but
> >>>> I
> >>>>>>>>> think an issue is that content is scattered in many
different
> >>>>>>>>> locations, and some of the docs are huge reference
guides where
> >>>> it
> >>>>>> can
> >>>>>>>>> be hard to find all the pieces of what you are trying
to do.
> >>>>>>>>>
> >>>>>>>>> The first thing a new contributor wants to do is
get the code
> >>>> and run
> >>>>>>>>> a build, and we do have a quick-start guide linked
to on the
> >>>> site,
> >>>>>> but
> >>>>>>>>> I think there is a lot of extra information in there
that is not
> >>>>>>>>> really relevant to someone just wanting get the
code and build.
> >>>> We
> >>>>>>>>> could have separate guides per OS like "Build NiFi
on Linux",
> >>>> "Build
> >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5
steps like:
> >>>>>>>>>
> >>>>>>>>> - Clone repo
> >>>>>>>>> - checkout master
> >>>>>>>>> - run maven
> >>>>>>>>> - cd to assembly
> >>>>>>>>> - ./bin/nifi.sh
> >>>>>>>>>
> >>>>>>>>> The next thing they want to do is contribute a change,
and we
> >>>> have a
> >>>>>>>>> great contributor guide, but again I think there
could be a very
> >>>>>> short
> >>>>>>>>> tutorial for the most common steps:
> >>>>>>>>>
> >>>>>>>>> - fork repo
> >>>>>>>>> - clone fork
> >>>>>>>>> - create branch
> >>>>>>>>> - make changes
> >>>>>>>>> - push branch
> >>>>>>>>> - submit pr
> >>>>>>>>>
> >>>>>>>>> and then say something like "for a more detailed
description of
> >>>> the
> >>>>>>>>> contribution process, please reference the Contributor
Guide".
> >>>>>>>>>
> >>>>>>>>> If we then make these getting started guides more
prominent
> >>>> right in
> >>>>>>>>> the middle of the NiFi homepage, then maybe they
will be easier
> >>>> to
> >>>>>>>>> find for new community members.
> >>>>>>>>>
> >>>>>>>>> We can keep extending this idea to other common
tasks beyond just
> >>>>>>>>> building and contributing.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> >>>> alopresto@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi folks,
> >>>>>>>>>>
> >>>>>>>>>> Based on some recent (and long-term) experiences,
I wanted to
> >>>>>> discuss
> >>>>>>>>> with the community what we could do to lower the
barrier of
> >>>> entry to
> >>>>>>>> using
> >>>>>>>>> & contributing to NiFi. I hope to get some good
feedback from
> >>>> both
> >>>>>>>>> long-time and newer members, and determine some
immediate
> >>>> concrete
> >>>>>>> steps
> >>>>>>>> we
> >>>>>>>>> can take.
> >>>>>>>>>>
> >>>>>>>>>> Problems identified:
> >>>>>>>>>> * NiFi has a number of custom profiles, so a
simple “mvn clean
> >>>>>>> install”
> >>>>>>>>> in project root doesn’t get a new developer up
and running
> >>>>>> immediately
> >>>>>>>>>> * The API is very well defined, but for new
contributors, it
> >>>> can
> >>>>>> be a
> >>>>>>>>> challenge to know where to put functionality, and
building a
> >>>> custom
> >>>>>>>>> processor + NAR and deploying isn’t a one-step
process
> >>>>>>>>>> * Project size (and build size/time) is large.
This can
> >>>> restrict
> >>>>>> the
> >>>>>>>>> minimum hardware necessary, elongate the development
cycle, etc.
> >>>>>>>>>> * Some new users do not receive mailing list
replies
> >>>>>>>>>>
> >>>>>>>>>> Possible solutions:
> >>>>>>>>>> * On a clean git clone, “mvn clean install”
should build a
> >>>> working
> >>>>>>>>> instance. Maybe we provide a quickstart.sh script
to handle the
> >>>>>> default
> >>>>>>>>> maven build, change to the target directory, and
start NiFi?
> >>>>>>>>>> * Individual contributors have written excellent
blogs, and
> >>>>>>>>> documentation exists, but making it more prominent
or more easily
> >>>>>>>> accessed
> >>>>>>>>> could help?
> >>>>>>>>>> * Extension registry will solve all the world’s
problems
> >>>> (related
> >>>>>> to
> >>>>>>>>> bundling and build time)
> >>>>>>>>>> * Not sure about this one — I don’t know
if it’s because
> >>>> they’re
> >>>>>> not
> >>>>>>>>> subscribed, their mail client is blocking them,
etc.
> >>>>>>>>>>
> >>>>>>>>>> I’ve said my bit, now I am eager to hear from
other community
> >>>>>> members
> >>>>>>>> on
> >>>>>>>>> their experiences, steps that helped them, and suggestions
for
> >>>> the
> >>>>>>> future
> >>>>>>>>> to continue to make the NiFi community welcoming
to new users.
> >>>>>> Thanks.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Andy LoPresto
> >>>>>>>>>> alopresto@apache.org
> >>>>>>>>>> alopresto.apache@gmail.com
> >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE
3C6E F65B 2F7D
> >>>> EF69
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
>

Mime
View raw message