nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan H <ryan.howell.developm...@gmail.com>
Subject Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control
Date Mon, 07 May 2018 12:29:57 GMT
Hi Kevin,

Thanks for the multiple options as it relates to FDLC; good articles. The
workaround for our current error was to manually search/destroy all
references to the registry in the created template xml file
(<versionControlInformation> tag in the template). Then when the template
is imported into a the new environment, the issue was resolved as all of
the bad references were not ported into the new environment.

It seems that the version control information (registry references)
shouldn't be included when creating templates or there should be an option
to include/exclude when creating the template. I would guess that there are
a lot of use cases where a template is being created to be used in
environments other than the one which it was created in. While it may not
fully align with best practices/suggested methods for FDLC, this does cause
issues if done this way. It may just be the case that this is documented as
being and issue and the recommendation is to follow one of the suggested
practices to avoid issues such as this one.

Cheers,

Ryan

On Mon, May 7, 2018 at 7:10 AM, Kevin Doran <kdoran@apache.org> wrote:

> Hi Ryan,
>
>
>
> I’ve never tried creating a template from a process group that was saved
> to a NiFi Registry, so I haven’t run into this exact error. However, there
> are users that cannot connect multiple environments (e.g., dev and
> production) to the same NiFi Registry and therefore have the need you’ve
> described of having to move flow definitions without a shared NiFi Registry
> instance.
>
>
>
> From what you’ve described and the error you’ve encountered, I’ve gathered
> that the approach you are using to migrate flows across environments is:
>
>
>
> 1. [Env A] Design / test a flow in NiFi, saving/loading from environment’s
> NiFi Registry as needed
>
> 2. [Env A] Export to template
>
> 3. [Env B] Import template into NiFI
>
> 4. [Env B] Save to environment’s NiFi Registry
>
>
>
> Is that more or less correct? If so, instead of this workflow, would it be
> possible to take the following path to migrate the flow version, which I
> have seen others use:
>
>
>
> [Env A] NiFi  >> [Env A] NiFi Registry >> [Env B] NiFi Registry >>
[Env B]
> NiFi
>
>
>
> In this FDLC workflow, templates are never used, because in Env B the
> Registry is populated first and then imported into NiFi using the import
> from Registry feature.
>
>
>
> If this sounds like it could work for your use case, and you would be able
> to avoid templates entirely (i.e., there is no other reason you need them
> other than migrating flows across environments), then there are lots of
> great write ups by others on how to achieve this workflow. The trick is in
> the Registry A to Registry B step, and for that there are a few approaches
> and tools that others have documented.
>
>
>
> A great write up about this “one Registry per environment” FDLC approach
> is included in a blog post by Pierre Villard, “Automate workflow deployment
> in Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can
> use the NiFi CLI [1], included in the NiFi Toolkit as a separate executable
> since NiFi 1.6.0 [2] as a tool to migrate flow version snapshots from one
> NiFi Registry to another. Specifically, the CLI tools “registry
> export-flow-version” and “registry import-flow-version” commands.
>
>
>
> An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi client
> SDK that makes the REST API for NiFi and NiFi Registry easier to work with
> from Python automation scripts. There’s a good write up on using this to
> interact with flows in NiFi Registry instances from Timothy Spann [5].
>
>
>
> I hope this helps you workaround the issue you are running into. It
> certainly does not solve the specific error you have documented, but
> perhaps it is a viable alternative that avoids that exception.
>
>
>
> Best,
> Kevin
>
>
>
> [1] https://pierrevillard.com/2018/04/09/automate-workflow-
> deployment-in-apache-nifi-with-the-nifi-registry/
>
> [2] https://github.com/apache/nifi/tree/master/nifi-toolkit/
> nifi-toolkit-cli
>
> [3] https://nifi.apache.org/download.html
>
> [4] https://github.com/Chaffelson/nipyapi
>
> [5] https://community.hortonworks.com/articles/177301/big-data-
> devops-apache-nifi-flow-versioning-and-au.html
>
>
>
>
>
> *From: *Ryan H <ryan.howell.development@gmail.com>
> *Reply-To: *<users@nifi.apache.org>
> *Date: *Sunday, May 6, 2018 at 21:05
> *To: *<users@nifi.apache.org>
> *Subject: *Error Reference When Creating Template From PG Under NiFi
> Registry Version Control
>
>
>
> Hi All,
>
> We have found what seems to be an issue when creating a template of a
> Process Group that is under version control with NiFi Registry. What seems
> to happen is that there is a hard reference to the Registry Instance from
> the Env that the template was created from within the template xml file.
> When importing the template into a new Env the reference to the registry
> goes with the template and looks to stay with the new process group, is
> written to the flow.xml.gz, and causes an issue when clustered (the nodes
> end up not being able to sync after a while because of the error and goes
> into a disconnected state).
>
> I'll say upfront that this somewhat defeats the purpose of having the
> registry and I'm guessing that there will be suggestions that we should
> either go the template FLDC route or go the registry FLDC route. For this
> purpose we cannot share the registry instance between the two environments
> and may be the case that we should not use the registry at this time.
>
> Should this be an error? Is a better way other than going in a
> searching/destroying all the old registry references out of the template
> and flow xml files? Should we manually change the registry instances in
> each of the environments to be the same guid id?
>
>
>
> As always any help or direction is greatly appreciated.
>
> Cheers,
>
> Ryan Howell
>
>
> template.xml snippet (created template to download, which will then be
> uploaded into new env) (seems that the error is automatically created
> knowing that it is going to be a problem importing):
> ...
> <versionControlInformation>
>                         <bucketId>1f26b7fd-8987-491d-
> a2ac-4f87292ab29b</bucketId>
>                         <bucketName>Dev-Test</bucketName>
>                         <flowDescription></flowDescription>
>                         <flowId>c8e54d77-ce35-4be7-
> aff6-339a96b58e7d</flowId>
>                         <flowName>MySuperCoolFlow</flowName>
>                         <registryId>4ee39816-0161-1000-ffff-ffffc4085349</
> registryId>
>                         <registryName>4ee39816-0161-
> 1000-ffff-ffffc4085349</registryName>
>                         <state>SYNC_FAILURE</state>
>                         <stateExplanation>Unable to synchronize Process
> Group with Flow Registry because Process Group was placed under Version
> Control using Flow Registry with identifier 4ee39816-0161-1000-ffff-ffffc4085349
> but cannot find any Flow Registry with this identifier</stateExplanation>
>                         <version>1</version>
>                     </versionControlInformation>
>                 </processGroups>
>                 <processGroups>
>                     <id>9d90f6ea-f2a1-3d65-0000-000000000000</id>
>                     <parentGroupId>d8cf7c44-48c7-3c7d-0000-000000000000</
> parentGroupId>
>                     <position>
>                         <x>422.23612851041094</x>
>                         <y>-1527.1966721061617</y>
>                     </position>
>                     <versionedComponentId>322cb8f0-cec6-30f9-9565-
> 6948120c96d6</versionedComponentId>
>                     <comments></comments>
>                     <contents>
>                         <connections>
>                             <id>80c3a35a-27ec-33a2-0000-000000000000</id>
>                             <parentGroupId>9d90f6ea-f2a1-
> 3d65-0000-000000000000</parentGroupId>
>                             <versionedComponentId>d77ff912-6127-39f2-9c7a-
> 2eb3c7dde817</versionedComponentId>
>                             <backPressureDataSizeThreshold>1 GB</
> backPressureDataSizeThreshold>
>                             <backPressureObjectThreshold>10000</
> backPressureObjectThreshold>
>                             <destination>
> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId>
> <id>14b964e1-0775-3e6f-0000-000000000000</id>
> <type>PROCESSOR</type>
> <versionedComponentId>66873d14-b329-3322-b845-d144571ac93e</
> versionedComponentId>
>                             </destination>
>                             <flowFileExpiration>0 sec</flowFileExpiration>
>                             <labelIndex>1</labelIndex>
>                             <name></name>
>                             <selectedRelationships>success</
> selectedRelationships>
>                             <source>
> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId>
> <id>f26999d5-adbe-33ea-0000-000000000000</id>
> <type>PROCESSOR</type>
> <versionedComponentId>c928e355-f4b0-3673-ba8e-9d7f0cf57d5f</
> versionedComponentId>
>                             </source>
>
>

Mime
View raw message