nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Srinivasan <james.sriniva...@gmail.com>
Subject Re: Lowering the barrier of entry
Date Fri, 25 Jan 2019 21:59:36 GMT
9) Oh, and the wiki is a little hard to navigate and the contents rather patchy

On Fri, 25 Jan 2019 at 21:57, James Srinivasan
<james.srinivasan@gmail.com> wrote:
>
> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> realise I could and possibly should submit PRs :)
>
> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> it would have been nice it if set up unit tests for me.
> 2) The User and Developer documentation is great and comprehensive.
> Finding the developer docs is a little painful (handful of items at
> the end of a scrolling list of 200+ processors)
> 3) The Developer docs could possibly do with a little more clarity on
> processor lifetime i.e. what is called when ^h^h^h - skimming back
> over the docs, it looks pretty clear now
> 4) Some example code for common operations e.g. getting/setting
> attributes or reading/writing/modifying flowfile content would be
> great.
> 5) When using existing processors for inspiration, best practices
> weren't always clear e.g. some generated properties inside
> getSupportedPropertyDescriptors(), others generated a private static
> list on init and returned that. Such differences are inevitable in a
> large project, but it would be nice to have something blessed to start
> from.
> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen)
> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> the great blog posts by Matt Burgess and others were invaluable
> 8) In case this all sounds too negative, NiFi is fab!
>
> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <aperepel@gmail.com> wrote:
> >
> > I am not against the archetype. But we need to spell out every step of the
> > way. I'd like to see a user thinking about their custom logic ASAP rather
> > than fighting the tools to get started. Those steps should be brain-dead,
> > just reflexes, if you know what I mean. Hell, let them create a custom
> > processor project or prototype in a script by accident even! :)
> >
> > On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <bbende@gmail.com> wrote:
> >
> > > That makes sense about the best practice for deploying to an
> > > additional lib directory.
> > >
> > > So for the project structure you are saying it would be easier to have
> > > a repo somewhere with essentially the same thing that is in the
> > > archetype, but they just clone it and rename it themselves (what the
> > > archetype does for you)?
> > >
> > > Something that I think would be awesome is if we could provide a
> > > web-based project initializer that would essentially run the archetype
> > > behind the scenes and then let you download the archive of the code,
> > > just like the spring-boot starter [1]. Not sure if their initializr is
> > > something that can be re-used and customized [2].
> > >
> > > The problem is we would need to host that somewhere.
> > >
> > > [1] https://start.spring.io/
> > > [2] https://github.com/spring-io/initializr
> > >
> > > On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <aperepel@gmail.com> wrote:
> > > >
> > > > We assume they create new projects from archetypes every day. They don't.
> > > >
> > > > We also assume they know how to deploy new NARs. Most don't. Especially
> > > if
> > > > we want them to follow best practices and create an additional NAR
> > > bundles
> > > > directory entry im the config (vs dumping into nifi lib).
> > > >
> > > > I can attest that I feel a bit lost myself every time I need to come back
> > > > to this and refresh my brain synapses. If we could make these not require
> > > > any of that and make simple thinga dead simple....
> > > >
> > > > Andrew
> > > >
> > > > On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <bbende@gmail.com> wrote:
> > > >
> > > > > Andrew,
> > > > >
> > > > > I'm not disagreeing with your points, but I'm curious how you see
> > > > > those two ideas being different from the processor archetype and
the
> > > > > wiki page with the archetype commands?
> > > > >
> > > > > Is it just that people don't know about it?
> > > > >
> > > > > -Bryan
> > > > >
> > > > > [1]
> > > > >
> > > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > > > >
> > > > > On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <ottobackwards@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > I think this ties into my other discuss thread on refreshing
the
> > > > > archetypes
> > > > > >
> > > > > >
> > > > > > On January 25, 2019 at 11:50:10, Andrew Grande (aperepel@gmail.com)
> > > > > wrote:
> > > > > >
> > > > > > I consistently see my users struggling when they move up the
nifi
> > > food
> > > > > > chain and start looking at custom processors. The good content
about
> > > > > > prototyping processsors via scripting processors and finalizing
with
> > > a
> > > > > full
> > > > > > NAR bundle is everywhere but where it should be.
> > > > > >
> > > > > > A few simple changes could help (not *more* docs). They are
great,
> > > much
> > > > > > better than in many other projecta, but people are already drowning
> > > in
> > > > > > those.
> > > > > >
> > > > > > How about:
> > > > > >
> > > > > > + ISP has a pre-populated processor sceleton. A simple no-op
to fill
> > > in
> > > > > is
> > > > > > miles better than a blank text area (which invokes a blank stare).
> > > > > >
> > > > > > + As much as we may loook down on this, but... A simple guide
to a
> > > full
> > > > > NAR
> > > > > > build as a series of copy/paste commands.
> > > > > >
> > > > > > There's more, but this should fit the context for now.
> > > > > >
> > > > > > Andrew
> > > > > >
> > > > > > On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <mikerthomsen@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > One of the changes we should make is to create a separate
guide for
> > > > > > product
> > > > > > > vendors on how to build and maintain a bundle. We're at
that point
> > > > > where
> > > > > > > vendors will have to do it on their own as extension providers,
so
> > > it
> > > > > > would
> > > > > > > be very helpful for them to have a simple and straight
forward
> > > document
> > > > > > > showing them what should be there, best practices for
> > > maintainability
> > > > > and
> > > > > > > where to announce it.
> > > > > > >
> > > > > > > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <bbende@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > I think we have a lot more documentation than most
projects, but
> > > I
> > > > > > > > think an issue is that content is scattered in many
different
> > > > > > > > locations, and some of the docs are huge reference
guides where
> > > it
> > > > > can
> > > > > > > > be hard to find all the pieces of what you are trying
to do.
> > > > > > > >
> > > > > > > > The first thing a new contributor wants to do is get
the code
> > > and run
> > > > > > > > a build, and we do have a quick-start guide linked
to on the
> > > site,
> > > > > but
> > > > > > > > I think there is a lot of extra information in there
that is not
> > > > > > > > really relevant to someone just wanting get the code
and build.
> > > We
> > > > > > > > could have separate guides per OS like "Build NiFi
on Linux",
> > > "Build
> > > > > > > > NiFi on Windows", etc, where each guide was 4-5 steps
like:
> > > > > > > >
> > > > > > > > - Clone repo
> > > > > > > > - checkout master
> > > > > > > > - run maven
> > > > > > > > - cd to assembly
> > > > > > > > - ./bin/nifi.sh
> > > > > > > >
> > > > > > > > The next thing they want to do is contribute a change,
and we
> > > have a
> > > > > > > > great contributor guide, but again I think there could
be a very
> > > > > short
> > > > > > > > tutorial for the most common steps:
> > > > > > > >
> > > > > > > > - fork repo
> > > > > > > > - clone fork
> > > > > > > > - create branch
> > > > > > > > - make changes
> > > > > > > > - push branch
> > > > > > > > - submit pr
> > > > > > > >
> > > > > > > > and then say something like "for a more detailed description
of
> > > the
> > > > > > > > contribution process, please reference the Contributor
Guide".
> > > > > > > >
> > > > > > > > If we then make these getting started guides more
prominent
> > > right in
> > > > > > > > the middle of the NiFi homepage, then maybe they will
be easier
> > > to
> > > > > > > > find for new community members.
> > > > > > > >
> > > > > > > > We can keep extending this idea to other common tasks
beyond just
> > > > > > > > building and contributing.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> > > alopresto@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi folks,
> > > > > > > > >
> > > > > > > > > Based on some recent (and long-term) experiences,
I wanted to
> > > > > discuss
> > > > > > > > with the community what we could do to lower the barrier
of
> > > entry to
> > > > > > > using
> > > > > > > > & contributing to NiFi. I hope to get some good
feedback from
> > > both
> > > > > > > > long-time and newer members, and determine some immediate
> > > concrete
> > > > > > steps
> > > > > > > we
> > > > > > > > can take.
> > > > > > > > >
> > > > > > > > > Problems identified:
> > > > > > > > > * NiFi has a number of custom profiles, so a
simple “mvn clean
> > > > > > install”
> > > > > > > > in project root doesn’t get a new developer up and
running
> > > > > immediately
> > > > > > > > > * The API is very well defined, but for new contributors,
it
> > > can
> > > > > be a
> > > > > > > > challenge to know where to put functionality, and
building a
> > > custom
> > > > > > > > processor + NAR and deploying isn’t a one-step process
> > > > > > > > > * Project size (and build size/time) is large.
This can
> > > restrict
> > > > > the
> > > > > > > > minimum hardware necessary, elongate the development
cycle, etc.
> > > > > > > > > * Some new users do not receive mailing list
replies
> > > > > > > > >
> > > > > > > > > Possible solutions:
> > > > > > > > > * On a clean git clone, “mvn clean install”
should build a
> > > working
> > > > > > > > instance. Maybe we provide a quickstart.sh script
to handle the
> > > > > default
> > > > > > > > maven build, change to the target directory, and start
NiFi?
> > > > > > > > > * Individual contributors have written excellent
blogs, and
> > > > > > > > documentation exists, but making it more prominent
or more easily
> > > > > > > accessed
> > > > > > > > could help?
> > > > > > > > > * Extension registry will solve all the world’s
problems
> > > (related
> > > > > to
> > > > > > > > bundling and build time)
> > > > > > > > > * Not sure about this one — I don’t know
if it’s because
> > > they’re
> > > > > not
> > > > > > > > subscribed, their mail client is blocking them, etc.
> > > > > > > > >
> > > > > > > > > I’ve said my bit, now I am eager to hear from
other community
> > > > > members
> > > > > > > on
> > > > > > > > their experiences, steps that helped them, and suggestions
for
> > > the
> > > > > > future
> > > > > > > > to continue to make the NiFi community welcoming to
new users.
> > > > > Thanks.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Andy LoPresto
> > > > > > > > > alopresto@apache.org
> > > > > > > > > alopresto.apache@gmail.com
> > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE
3C6E F65B 2F7D
> > > EF69
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >

Mime
View raw message