nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Ladner <eric.lad...@gmail.com>
Subject Re: JSON handling question
Date Tue, 14 May 2019 02:55:33 GMT
Thanks for the suggestions, Matt and Andy.  I'll look into the
ExecuteScript.

Andy, I don't think that ExtractText or EvaluateJsonPath is robust enough
to pull the json from the record.

"Select A, B, C from some_table" returns records like:
'UUID FIELD', '01-JAN-15', '{ "attribute1" : "some value", "attribute2" :
"other value", "randomField" : "randomValue", ...}'

The embedded JSON can have anywhere between 5 and 30 attributes, usually
not in any particular order.

On Mon, May 13, 2019 at 5:53 PM Andy LoPresto <alopresto@apache.org> wrote:

> I think if you wanted to do this just using native processors, a
> convoluted way would be DuplicateFlowFile, ExtractText or EvaluateJsonPath
> to get the content into an attribute, and then ReplaceText to replace the
> (new) flowfile’s content with the value of the specified attribute.
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 13, 2019, at 3:20 PM, Matt Burgess <mattyb149@apache.org> wrote:
>
> Eric,
>
> AFAIK we don't have a way to convert a string field to a JSON object
> outside of something like an ExecuteScript processor. You can use
> QueryRecord to do a "SELECT C FROM FLOWFILE" or a ConvertRecord
> processor with a writer that only includes the C field (the rest will
> be removed), but the result will be a JSON string inside quotes. From
> there you can use ExecuteScript with something like Groovy that can
> take a string and parse a JSON object from it.  You could try a
> FreeFormRecordSetWriter but I still think the quotes come with the
> string field.
>
> With the advent of schema inference, we should take a look at
> something like an "asJSON()" function in QueryRecord which could infer
> the schema of the JSON object and merge that record schema into the
> overall outgoing record schema. Alternatively (or additionally) maybe
> we could add a ConvertField processor to do something similar.
>
> Regards,
> Matt
>
> On Mon, May 13, 2019 at 5:11 PM Eric Ladner <eric.ladner@gmail.com> wrote:
>
>
> (sorry if this is duplicated but it didn't like my work email address).
>
> I have an instance where I'm pulling data via the QueryDatabaseTable
> processor and one of the columns contains a valid JSON string that I'd like
> to split out into a separate flowfile to be handled separately.  I can't
> seem to find a way to peel it out into its own entity, though.
>
> As an example, "select A, B, C from some_table" returns rows of data where
> the text value in column "C" is valid JSON.  In a standard "SplitAvro ->
> AvroToJson" flow, $.C would be the JSON value from the select.  I just
> can't figure out how to promote the value of an attribute in the query to
> its own flowfile.
>
> I'm probably missing something simple, but any tips would be appreciated.
>
> --
> Eric Ladner
>
>
>

-- 
Eric Ladner

Mime
View raw message