nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan H <>
Subject Dynamic Mapping
Date Thu, 04 Oct 2018 17:21:55 GMT
Hi All,

I have been working on an integration between two systems with NiFi in the
middle as the integration point. Basically when an event happens on
System-A, such as a record update, those changes need to be propagated to
System-B. For the first use case, I have set up a data flow that listens
for incoming requests from System-A, performs a mapping, the sends the
mapped data to System-B.

Generalized Flow for "Create_Event" (dumbed down significantly):
System-A "Create_Event" -> HandleHTTPRequest -> JoltTransformJSON ->
InvokeHTTP -> System-B "Create_Event"

This works great for the first case with a predefined mapping in
JoltTransformJSON. Now I want to generalize it so that the same data flow
can be used for all Create_Event's on System-A.

Here is where the issue comes in. There is a base schema for System-A that
has a base mapping to the base schema in System-B. Users of the System have
the ability to "extend" the base schema to add/remove fields and modify the
base mapping. So each time the Create_Event happens, the mapping that is
used should be the unique mapping spec associated to that user (call it a
GUID that comes along with the request).

The data flow is the exact same for all Create_Events, except for the
mapping, which will be unique to the user.

Does anyone know of a way to load up a different mapping to be used on a
per-request basis? I used JoltTransformJSON just as a proof of concept, so
it does not need to be a Jolt spec and can be modified to meet the needs of
whatever would work for this.

I started to look into schema registry, but kind of got lost a bit and
wasn't sure if it could be applied to this situation.

Any Help is, as always, greatly appreciated.


Ryan H.

View raw message