nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?
Date Thu, 16 May 2019 19:35:23 GMT
Excellent Bryan. Looking forward to this release. Thanks for taking the initiative here. 

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 16, 2019, at 12:34 PM, Bryan Bende <bbende@gmail.com> wrote:
> 
> The last few items tagged for 0.4.0 have been merged so we should be
> good to put out an RC for 0.4.0.
> 
> I'll start working on getting everything together and try to get
> something out soon.
> 
> On Fri, May 10, 2019 at 9:00 AM Bryan Bende <bbende@gmail.com> wrote:
>> 
>> I think we should be able to kick 0.4.0 very soon, just a couple of
>> items to get in. Assuming those get reviewed and merged soon we could
>> possibly kick out an RC early next week. I can plan to be RM unless
>> anyone else is interested.
>> 
>> On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <bbende@gmail.com> wrote:
>>> 
>>> I think Kevin summarized it nicely. Auto-pulling the extensions with a
>>> versioned flow is definitely the ultimate vision, but we need to take
>>> steps to get there.
>>> 
>>> The process of automatically bringing in the extensions is something
>>> that would be implemented on the NiFi side and will require a bit of
>>> thought and design around the user experience.
>>> 
>>> Before we can do any of that, we first need somewhere to store and
>>> retrieve versioned extensions, which is the work that is underway on
>>> the NiFi Registry side.
>>> 
>>> On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <kdoran@apache.org> wrote:
>>>> 
>>>> Hi Ryan,
>>>> 
>>>> Bryan can speak more to this than I can, but, yes, providing the
>>>> experience you describe is the eventual goal! Aside from the NiFi
>>>> Extension Registry support though (the initial bits of which is what
>>>> Bryan is referring to in 0.4.0), getting the full, streamlined
>>>> experience in NiFi will require changes to NiFi as well, for example
>>>> in the logic to import a flow as it now also has to reach back to
>>>> Registry to pull extensions referenced by the flow during import. So
>>>> the changes to NiFi Registry to support hosting NiFi Extensions is
>>>> only half of the puzzle. This is why Bryan also mentioned it might be
>>>> worth holding back 0.4.0 (or releasing it, but marking the new
>>>> extension stuff as experimental), as it is likely the current
>>>> implementation and interfaces will need to change once the NiFi work
>>>> begins.
>>>> 
>>>> Cheers,
>>>> Kevin
>>>> 
>>>> On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
>>>> <ryan.andrew.hendrickson@gmail.com> wrote:
>>>>> 
>>>>> We'd like to connect a NiFi to a Registry, Pull down a Process Group,
and
>>>>> get any NARs required for that without having to manually install them
on
>>>>> the NiFi.
>>>>> 
>>>>> That's what I'm understanding the Extension part of the 0.4.0 release
to
>>>>> be, am I on point, or way off?
>>>>> 
>>>>> Thanks,
>>>>> Ryan
>>>>> 
>>>>> On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <bryanrosander@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey Pierre,
>>>>>> 
>>>>>> You can version the metadata db in the root dir of the git repo with
event
>>>>>> hooks.
>>>>>> 
>>>>>> With an h2 jdbc url of
>>>>>> 
>>>>>> 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
>>>>>> 
>>>>>> You can connect to the db in a separate process, we're using that
with a
>>>>>> shell event hook to dump the db to sql script when mutations happen
(now
>>>>>> tenant events too :) ), storing that in the git repo using
>>>>>> org.h2.tools.Script, and committing it.  Then we can restore from
that sql
>>>>>> script on startup before calling the stock startup script.
>>>>>> 
>>>>>> Side benefit is you can see metadata state changes in the git history
and
>>>>>> manually inspect the sql.
>>>>>> 
>>>>>> 
>>>>>> On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
>>>>>> pierre.villard.fr@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Bryan,
>>>>>>> 
>>>>>>> Thanks for the details. On my side, the main feature I (and others
I
>>>>>>> suppose) am looking for is the ability to rebuild the metadata
DB from
>>>>>> the
>>>>>>> Git repo. But to be honest, this is not a blocker at all: I can
live
>>>>>> with a
>>>>>>> custom Docker image containing this feature.
>>>>>>> 
>>>>>>> So, from my point of view, I don't have any issue with #2 but
maybe #1
>>>>>>> would also give access to experimental features and have interesting
>>>>>>> feedback from the community. But we would, indeed, need to make
it
>>>>>> crystal
>>>>>>> clear that the APIs could change in ulterior release(s).
>>>>>>> 
>>>>>>> Pierre
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Le mer. 27 mars 2019 à 18:11, Bryan Bende <bbende@gmail.com>
a écrit :
>>>>>>> 
>>>>>>>> Ryan,
>>>>>>>> 
>>>>>>>> The short answer to your question is no, there isn't actually
anything
>>>>>>>> new in the UI. The registry UI is setup in a generic way
to list
>>>>>>>> "versioned items", so versioned extension bundles will just
>>>>>>>> automatically show up in the list with versioned flows.
>>>>>>>> 
>>>>>>>> Let me put together a wiki page that outlines all of the
moving parts
>>>>>>>> for the extension registry work and where we are currently
at. I'll
>>>>>>>> send the link on this thread when I have something.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> For Pierre and others,
>>>>>>>> 
>>>>>>>> I was thinking more about this discussion of releasing 0.4.0,
and I
>>>>>>>> think there are two of approaches we could take...
>>>>>>>> 
>>>>>>>> 1) If there is a desire for the community to release 0.4.0
sooner
>>>>>>>> rather than later, then we can mark all of the extension
bundle
>>>>>>>> related APIs as experimental, which would then give us the
freedom to
>>>>>>>> continue evolving them until we have the full extension registry
>>>>>>>> experience, and likely mark them as stable in 0.5.0, or whichever
>>>>>>>> future release we decide. The reason for this is that there
will
>>>>>>>> likely be API changes needed while building out the NiFi
side of
>>>>>>>> things, and I don't want us to get stuck in a spot where
we can't
>>>>>>>> change some API since it has been released, and so far the
NiFi side
>>>>>>>> of the work hasn't been started yet.
>>>>>>>> 
>>>>>>>> 2) Wait until more of the extension registry work is finalized
and use
>>>>>>>> that as the driver of when to release 0.4.0, with the obvious
downside
>>>>>>>> that it may take a bit longer to get a release out which
then holds up
>>>>>>>> anything else that is not related to extension registry.
>>>>>>>> 
>>>>>>>> So I guess we just have to decide the driving factor for
releasing
>>>>>>>> 0.4.0... If its the non-extension related features/fixes,
then #1 is
>>>>>>>> probably the best approach. If its the extension related
stuff, then
>>>>>>>> I'd vote for #2 since it isn't really ready yet.
>>>>>>>> 
>>>>>>>> -Bryan
>>>>>>>> 
>>>>>>>> On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
>>>>>>>> <ryan.andrew.hendrickson@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Bryan,
>>>>>>>>>   Is there a new GUI part of the registry for the extension
registry
>>>>>>>> work
>>>>>>>>> in master?  I compiled 0.4.0-SNAPSHOT and didn't see
anything
>>>>>>>> immediately.
>>>>>>>>> You mentioned docs and sys admin stuff... Could you point
me in the
>>>>>>>>> quick-start path if I wanted to try to kick-the-tires
a little on
>>>>>> this?
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Ryan
>>>>>>>>> 
>>>>>>>>> On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bbende@gmail.com>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Mike,
>>>>>>>>>> 
>>>>>>>>>> All record-oriented components are extensions and
thus can make use
>>>>>>> of
>>>>>>>>>> versioned components (i.e. nothing related to records
is baked into
>>>>>>>>>> the core framework of NiFi).
>>>>>>>>>> 
>>>>>>>>>> The record API is essentially the module named
>>>>>>>>>> nifi-record-serialization-services-api which at runtime
is included
>>>>>>> in
>>>>>>>>>> nifi-standard-services-api-nar. Any record oriented
processors you
>>>>>>>>>> build are dependent on a specific version of
>>>>>>>>>> nifi-standard-services-api-nar since that is where
the Java
>>>>>> interface
>>>>>>>>>> will be that your processor was compiled against.
>>>>>>>>>> 
>>>>>>>>>> Right now, even without extension registry, you could
deploy
>>>>>> multiple
>>>>>>>>>> versions of nifi-standard-services-api-nar to a NiFi
instance, and
>>>>>>>>>> then deploy multiple versions of your record processing
NAR that
>>>>>>>>>> correspond to the versions of your API NAR.
>>>>>>>>>> 
>>>>>>>>>> Going forward with extension registry, we probably
want to consider
>>>>>>>>>> breaking up nifi-standard-services-api-nar into smaller
API NARs,
>>>>>> as
>>>>>>>>>> well as nifi-standard-bundle into smaller processor
bundles. For
>>>>>>>>>> example, there could be a record API NAR and a record
processors
>>>>>> NAR,
>>>>>>>>>> that would both be split out of from the standard
NARs.
>>>>>>>>>> 
>>>>>>>>>> -Bryan
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
>>>>>> mikerthomsen@gmail.com
>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Great news about the extension registry! As we
get closer to that
>>>>>>>> being
>>>>>>>>>>> ready, I'd like to add in some discussion about
versioning the
>>>>>>>> record API
>>>>>>>>>>> separately. A lot of the custom processors we
build now are
>>>>>> Record
>>>>>>>>>>> API-based, and it would be great to be able to
decouple them from
>>>>>>> one
>>>>>>>>>>> specific release of NiFi.
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bbende@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Pierre,
>>>>>>>>>>>> 
>>>>>>>>>>>> I think we are definitely close to an 0.4.0
release. A major
>>>>>>> chunk
>>>>>>>> of
>>>>>>>>>>>> extension registry work has already landed
in master, and I
>>>>>> still
>>>>>>>> have
>>>>>>>>>> one
>>>>>>>>>>>> other jira that I almost have ready and wanted
to include,
>>>>>>>> NIFIREG-233
>>>>>>>>>> for
>>>>>>>>>>>> generating extensions docs. Plus we probably
need to make a few
>>>>>>>>>>>> additions/updates to the admin guide.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Bryan
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard
<
>>>>>>>>>>>> pierre.villard.fr@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> All,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Some really nice features have been included
since the NiFi
>>>>>>>> Registry
>>>>>>>>>>>> 0.3.0
>>>>>>>>>>>>> release and I'm wondering if it'd be
a good time to consider
>>>>>> a
>>>>>>>> 0.4.0
>>>>>>>>>>>>> release.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There is a JIRA tagged for 0.4.0 and
there are 2 opened pull
>>>>>>>>>> requests,
>>>>>>>>>>>> plus
>>>>>>>>>>>>> interesting discussions / JIRAS on the
mailing lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Are there any reasons to hold off on
a 0.4.0 release?  Are
>>>>>>> there
>>>>>>>>>>>> particular
>>>>>>>>>>>>> JIRAs that the community considers necessary
to have as part
>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> release?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If not, happy to give it a try at performing
RM duties!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Pierre
>>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Sent from Gmail Mobile
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message