I am not entirely sure I fully understand the scenario so read the
following with a bit of caution.
But the gist I think i'm reading is that system A generates data
against a given schema referenceable with some guid say 'GUID1'. For
every GUID1 from System A there is a similar/but possibly different
schema in System B referenceable again by some guid say GUID1_B.
Are those base schemas you referenced say GUID1 and GUID1_B compatible
in that all fields of import in GUID1 schema will be in GUID1_B
schema? As-in can you treat GUID1_B schemas as a superset of GUID1
In any event, it is very possible that you can achieve this using the
Record processors and Record oriented controller services.
You can 'read' data from system A in NiFi using record readers that
reference schemas from systemA and 'write' data using system B schemas
The record oriented processors combined with their pluggable record
readers and writers allows this to happen. The schema for the reader
and writer can be different and the processors will map things over by
field names/types. That leaves the problem of 'how to access' the
schema information so NiFi knows what to do during read/write phases.
You can either ensure all these schemas are in the nifi schema
registry and can be looked up by some well established name and naming
scheme. You could also do it via a SQL lookup if you have the schemas
in some database (similar to what Carlos might be suggesting but
without Jolt). Or some other way..
Now, if your mapping logic from System A schema to System B schema is
more complex then you'll want something like JOLT most likely to help
manage those transforms.
You can also do some routing on the requests from System A to
specialized JOLT mappers for each case of converting to System B
schemas. That will end up in a lot of config but it is likely you can
paramaterize this well using versioned flows and expression langauge
statements in the variable registry.
On Thu, Oct 4, 2018 at 1:35 PM Ryan H <email@example.com> wrote:
> Hi All,
> I have been working on an integration between two systems with NiFi in the middle as the integration point. Basically when an event happens on System-A, such as a record update, those changes need to be propagated to System-B. For the first use case, I have set up a data flow that listens for incoming requests from System-A, performs a mapping, the sends the mapped data to System-B.
> Generalized Flow for "Create_Event" (dumbed down significantly):
> System-A "Create_Event" -> HandleHTTPRequest -> JoltTransformJSON -> InvokeHTTP -> System-B "Create_Event"
> This works great for the first case with a predefined mapping in JoltTransformJSON. Now I want to generalize it so that the same data flow can be used for all Create_Event's on System-A.
> Here is where the issue comes in. There is a base schema for System-A that has a base mapping to the base schema in System-B. Users of the System have the ability to "extend" the base schema to add/remove fields and modify the base mapping. So each time the Create_Event happens, the mapping that is used should be the unique mapping spec associated to that user (call it a GUID that comes along with the request).
> The data flow is the exact same for all Create_Events, except for the mapping, which will be unique to the user.
> Does anyone know of a way to load up a different mapping to be used on a per-request basis? I used JoltTransformJSON just as a proof of concept, so it does not need to be a Jolt spec and can be modified to meet the needs of whatever would work for this.
> I started to look into schema registry, but kind of got lost a bit and wasn't sure if it could be applied to this situation.
> Any Help is, as always, greatly appreciated.
> Ryan H.