nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: JSON handling question
Date Tue, 14 May 2019 03:05:38 GMT
Not sure if you need CSV for the alternate path in your flow, but if
not and you use QueryDatabaseTableRecord you can return JSON rather
than Avro. The nested JSON field will still be a string, but then you
could use EvaluateJsonPath followed by ReplaceText as Andy suggested.

On Mon, May 13, 2019 at 10:55 PM Eric Ladner <eric.ladner@gmail.com> wrote:
>
> Thanks for the suggestions, Matt and Andy.  I'll look into the ExecuteScript.
>
> Andy, I don't think that ExtractText or EvaluateJsonPath is robust enough to pull the
json from the record.
>
> "Select A, B, C from some_table" returns records like:
> 'UUID FIELD', '01-JAN-15', '{ "attribute1" : "some value", "attribute2" : "other value",
"randomField" : "randomValue", ...}'
>
> The embedded JSON can have anywhere between 5 and 30 attributes, usually not in any particular
order.
>
> On Mon, May 13, 2019 at 5:53 PM Andy LoPresto <alopresto@apache.org> wrote:
>>
>> I think if you wanted to do this just using native processors, a convoluted way would
be DuplicateFlowFile, ExtractText or EvaluateJsonPath to get the content into an attribute,
and then ReplaceText to replace the (new) flowfile’s content with the value of the specified
attribute.
>>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> alopresto.apache@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On May 13, 2019, at 3:20 PM, Matt Burgess <mattyb149@apache.org> wrote:
>>
>> Eric,
>>
>> AFAIK we don't have a way to convert a string field to a JSON object
>> outside of something like an ExecuteScript processor. You can use
>> QueryRecord to do a "SELECT C FROM FLOWFILE" or a ConvertRecord
>> processor with a writer that only includes the C field (the rest will
>> be removed), but the result will be a JSON string inside quotes. From
>> there you can use ExecuteScript with something like Groovy that can
>> take a string and parse a JSON object from it.  You could try a
>> FreeFormRecordSetWriter but I still think the quotes come with the
>> string field.
>>
>> With the advent of schema inference, we should take a look at
>> something like an "asJSON()" function in QueryRecord which could infer
>> the schema of the JSON object and merge that record schema into the
>> overall outgoing record schema. Alternatively (or additionally) maybe
>> we could add a ConvertField processor to do something similar.
>>
>> Regards,
>> Matt
>>
>> On Mon, May 13, 2019 at 5:11 PM Eric Ladner <eric.ladner@gmail.com> wrote:
>>
>>
>> (sorry if this is duplicated but it didn't like my work email address).
>>
>> I have an instance where I'm pulling data via the QueryDatabaseTable processor and
one of the columns contains a valid JSON string that I'd like to split out into a separate
flowfile to be handled separately.  I can't seem to find a way to peel it out into its own
entity, though.
>>
>> As an example, "select A, B, C from some_table" returns rows of data where the text
value in column "C" is valid JSON.  In a standard "SplitAvro -> AvroToJson" flow, $.C would
be the JSON value from the select.  I just can't figure out how to promote the value of an
attribute in the query to its own flowfile.
>>
>> I'm probably missing something simple, but any tips would be appreciated.
>>
>> --
>> Eric Ladner
>>
>>
>
>
> --
> Eric Ladner

Mime
View raw message