nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: Lowering the barrier of entry
Date Fri, 25 Jan 2019 23:04:16 GMT
James,

I’m wondering if a page outlining a toy processor (something like `CountText` or `ReverseContent`)
and doing a line-by-line annotation from a developer’s perspective would be helpful. It
could be a few sections:

1. How to get to this point
	* running the maven archetype
	* choosing the directory to install to
	* putting the class name in the manifest file
	* etc.
2. The code
	* here’s the class
	* here’s what extending AbstractProcessor gets you, etc. A lot of this is currently in
the Developer Guide, but in textbook mode
	* here’s how to modify some code (i.e. write one line of Java that switches it from straight
content pass-through to reversing the text)
	* here’s how to make a unit test (introduce the TestRunner framework, etc.)
3. Running, building, installing
	* Run your unit test from the IDE/maven
	* Build the NAR module
	* Install the NAR in NiFi lib/ or custom/
	* Restart NiFi
		* See the NAR loaded in the log
		* Deploy the component on the canvas

I imagine this being written more conversationally/blog-like than most of our current reference
documentation to be used as a split-screen walkthrough. Each section could certainly link
to the existing detailed documentation for various topics, like the processor lifecycle, etc.


Does this sounds like something that would have helped you?

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 25, 2019, at 1:59 PM, James Srinivasan <james.srinivasan@gmail.com> wrote:
> 
> 9) Oh, and the wiki is a little hard to navigate and the contents rather patchy
> 
> On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> <james.srinivasan@gmail.com> wrote:
>> 
>> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
>> realise I could and possibly should submit PRs :)
>> 
>> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
>> it would have been nice it if set up unit tests for me.
>> 2) The User and Developer documentation is great and comprehensive.
>> Finding the developer docs is a little painful (handful of items at
>> the end of a scrolling list of 200+ processors)
>> 3) The Developer docs could possibly do with a little more clarity on
>> processor lifetime i.e. what is called when ^h^h^h - skimming back
>> over the docs, it looks pretty clear now
>> 4) Some example code for common operations e.g. getting/setting
>> attributes or reading/writing/modifying flowfile content would be
>> great.
>> 5) When using existing processors for inspiration, best practices
>> weren't always clear e.g. some generated properties inside
>> getSupportedPropertyDescriptors(), others generated a private static
>> list on init and returned that. Such differences are inevitable in a
>> large project, but it would be nice to have something blessed to start
>> from.
>> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen)
>> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
>> the great blog posts by Matt Burgess and others were invaluable
>> 8) In case this all sounds too negative, NiFi is fab!
>> 
>> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <aperepel@gmail.com> wrote:
>>> 
>>> I am not against the archetype. But we need to spell out every step of the
>>> way. I'd like to see a user thinking about their custom logic ASAP rather
>>> than fighting the tools to get started. Those steps should be brain-dead,
>>> just reflexes, if you know what I mean. Hell, let them create a custom
>>> processor project or prototype in a script by accident even! :)
>>> 
>>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <bbende@gmail.com> wrote:
>>> 
>>>> That makes sense about the best practice for deploying to an
>>>> additional lib directory.
>>>> 
>>>> So for the project structure you are saying it would be easier to have
>>>> a repo somewhere with essentially the same thing that is in the
>>>> archetype, but they just clone it and rename it themselves (what the
>>>> archetype does for you)?
>>>> 
>>>> Something that I think would be awesome is if we could provide a
>>>> web-based project initializer that would essentially run the archetype
>>>> behind the scenes and then let you download the archive of the code,
>>>> just like the spring-boot starter [1]. Not sure if their initializr is
>>>> something that can be re-used and customized [2].
>>>> 
>>>> The problem is we would need to host that somewhere.
>>>> 
>>>> [1] https://start.spring.io/
>>>> [2] https://github.com/spring-io/initializr
>>>> 
>>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <aperepel@gmail.com>
wrote:
>>>>> 
>>>>> We assume they create new projects from archetypes every day. They don't.
>>>>> 
>>>>> We also assume they know how to deploy new NARs. Most don't. Especially
>>>> if
>>>>> we want them to follow best practices and create an additional NAR
>>>> bundles
>>>>> directory entry im the config (vs dumping into nifi lib).
>>>>> 
>>>>> I can attest that I feel a bit lost myself every time I need to come
back
>>>>> to this and refresh my brain synapses. If we could make these not require
>>>>> any of that and make simple thinga dead simple....
>>>>> 
>>>>> Andrew
>>>>> 
>>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <bbende@gmail.com> wrote:
>>>>> 
>>>>>> Andrew,
>>>>>> 
>>>>>> I'm not disagreeing with your points, but I'm curious how you see
>>>>>> those two ideas being different from the processor archetype and
the
>>>>>> wiki page with the archetype commands?
>>>>>> 
>>>>>> Is it just that people don't know about it?
>>>>>> 
>>>>>> -Bryan
>>>>>> 
>>>>>> [1]
>>>>>> 
>>>> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>>>>>> 
>>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <ottobackwards@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> I think this ties into my other discuss thread on refreshing
the
>>>>>> archetypes
>>>>>>> 
>>>>>>> 
>>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande (aperepel@gmail.com)
>>>>>> wrote:
>>>>>>> 
>>>>>>> I consistently see my users struggling when they move up the
nifi
>>>> food
>>>>>>> chain and start looking at custom processors. The good content
about
>>>>>>> prototyping processsors via scripting processors and finalizing
with
>>>> a
>>>>>> full
>>>>>>> NAR bundle is everywhere but where it should be.
>>>>>>> 
>>>>>>> A few simple changes could help (not *more* docs). They are great,
>>>> much
>>>>>>> better than in many other projecta, but people are already drowning
>>>> in
>>>>>>> those.
>>>>>>> 
>>>>>>> How about:
>>>>>>> 
>>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op
to fill
>>>> in
>>>>>> is
>>>>>>> miles better than a blank text area (which invokes a blank stare).
>>>>>>> 
>>>>>>> + As much as we may loook down on this, but... A simple guide
to a
>>>> full
>>>>>> NAR
>>>>>>> build as a series of copy/paste commands.
>>>>>>> 
>>>>>>> There's more, but this should fit the context for now.
>>>>>>> 
>>>>>>> Andrew
>>>>>>> 
>>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <mikerthomsen@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> One of the changes we should make is to create a separate
guide for
>>>>>>> product
>>>>>>>> vendors on how to build and maintain a bundle. We're at that
point
>>>>>> where
>>>>>>>> vendors will have to do it on their own as extension providers,
so
>>>> it
>>>>>>> would
>>>>>>>> be very helpful for them to have a simple and straight forward
>>>> document
>>>>>>>> showing them what should be there, best practices for
>>>> maintainability
>>>>>> and
>>>>>>>> where to announce it.
>>>>>>>> 
>>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <bbende@gmail.com>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> I think we have a lot more documentation than most projects,
but
>>>> I
>>>>>>>>> think an issue is that content is scattered in many different
>>>>>>>>> locations, and some of the docs are huge reference guides
where
>>>> it
>>>>>> can
>>>>>>>>> be hard to find all the pieces of what you are trying
to do.
>>>>>>>>> 
>>>>>>>>> The first thing a new contributor wants to do is get
the code
>>>> and run
>>>>>>>>> a build, and we do have a quick-start guide linked to
on the
>>>> site,
>>>>>> but
>>>>>>>>> I think there is a lot of extra information in there
that is not
>>>>>>>>> really relevant to someone just wanting get the code
and build.
>>>> We
>>>>>>>>> could have separate guides per OS like "Build NiFi on
Linux",
>>>> "Build
>>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps
like:
>>>>>>>>> 
>>>>>>>>> - Clone repo
>>>>>>>>> - checkout master
>>>>>>>>> - run maven
>>>>>>>>> - cd to assembly
>>>>>>>>> - ./bin/nifi.sh
>>>>>>>>> 
>>>>>>>>> The next thing they want to do is contribute a change,
and we
>>>> have a
>>>>>>>>> great contributor guide, but again I think there could
be a very
>>>>>> short
>>>>>>>>> tutorial for the most common steps:
>>>>>>>>> 
>>>>>>>>> - fork repo
>>>>>>>>> - clone fork
>>>>>>>>> - create branch
>>>>>>>>> - make changes
>>>>>>>>> - push branch
>>>>>>>>> - submit pr
>>>>>>>>> 
>>>>>>>>> and then say something like "for a more detailed description
of
>>>> the
>>>>>>>>> contribution process, please reference the Contributor
Guide".
>>>>>>>>> 
>>>>>>>>> If we then make these getting started guides more prominent
>>>> right in
>>>>>>>>> the middle of the NiFi homepage, then maybe they will
be easier
>>>> to
>>>>>>>>> find for new community members.
>>>>>>>>> 
>>>>>>>>> We can keep extending this idea to other common tasks
beyond just
>>>>>>>>> building and contributing.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
>>>> alopresto@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi folks,
>>>>>>>>>> 
>>>>>>>>>> Based on some recent (and long-term) experiences,
I wanted to
>>>>>> discuss
>>>>>>>>> with the community what we could do to lower the barrier
of
>>>> entry to
>>>>>>>> using
>>>>>>>>> & contributing to NiFi. I hope to get some good feedback
from
>>>> both
>>>>>>>>> long-time and newer members, and determine some immediate
>>>> concrete
>>>>>>> steps
>>>>>>>> we
>>>>>>>>> can take.
>>>>>>>>>> 
>>>>>>>>>> Problems identified:
>>>>>>>>>> * NiFi has a number of custom profiles, so a simple
“mvn clean
>>>>>>> install”
>>>>>>>>> in project root doesn’t get a new developer up and
running
>>>>>> immediately
>>>>>>>>>> * The API is very well defined, but for new contributors,
it
>>>> can
>>>>>> be a
>>>>>>>>> challenge to know where to put functionality, and building
a
>>>> custom
>>>>>>>>> processor + NAR and deploying isn’t a one-step process
>>>>>>>>>> * Project size (and build size/time) is large. This
can
>>>> restrict
>>>>>> the
>>>>>>>>> minimum hardware necessary, elongate the development
cycle, etc.
>>>>>>>>>> * Some new users do not receive mailing list replies
>>>>>>>>>> 
>>>>>>>>>> Possible solutions:
>>>>>>>>>> * On a clean git clone, “mvn clean install” should
build a
>>>> working
>>>>>>>>> instance. Maybe we provide a quickstart.sh script to
handle the
>>>>>> default
>>>>>>>>> maven build, change to the target directory, and start
NiFi?
>>>>>>>>>> * Individual contributors have written excellent
blogs, and
>>>>>>>>> documentation exists, but making it more prominent or
more easily
>>>>>>>> accessed
>>>>>>>>> could help?
>>>>>>>>>> * Extension registry will solve all the world’s
problems
>>>> (related
>>>>>> to
>>>>>>>>> bundling and build time)
>>>>>>>>>> * Not sure about this one — I don’t know if it’s
because
>>>> they’re
>>>>>> not
>>>>>>>>> subscribed, their mail client is blocking them, etc.
>>>>>>>>>> 
>>>>>>>>>> I’ve said my bit, now I am eager to hear from other
community
>>>>>> members
>>>>>>>> on
>>>>>>>>> their experiences, steps that helped them, and suggestions
for
>>>> the
>>>>>>> future
>>>>>>>>> to continue to make the NiFi community welcoming to new
users.
>>>>>> Thanks.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Andy LoPresto
>>>>>>>>>> alopresto@apache.org
>>>>>>>>>> alopresto.apache@gmail.com
>>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E
F65B 2F7D
>>>> EF69
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message