nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lone Pine Account <lp...@roobricks.com>
Subject Re: Strange behavior with PutSQL
Date Wed, 15 Aug 2018 15:16:50 GMT
That was it!  The fragmented transactions settings were not correct.

Thanks a million!

On Wed, Aug 15, 2018 at 11:12 AM, Matt Burgess <mattyb149@apache.org> wrote:

> Do you have a Split processor upstream? If so, is the setting of
> Support Fragmented Transactions in PutSQL set to true? That
> combination will have PutSQL try to find all the flow files with the
> same "fragment.id" attribute, and will only proceed if it gets
> "fragment.count" of them (all the flow files with the same fragment.id
> should have the same fragment.count value, PutSQL checks for this and
> should log an error/warning if not). In order to get that to work, the
> Batch Size property would have to be at least as big as what you
> expect your fragment.count to be (perhaps twice the size). You may
> also want to use FirstInFirstOutPrioritizer on all the connections
> downstream from the Split.
>
> Alternatively, if the order of the incoming SQL statements doesn't
> matter, set Support Fragmented Transactions to false.
>
> Having said all that, unless you have a reason to generate your own
> SQL, consider using PutDatabaseRecord rather than Split -> Replace ->
> PutSQL, it handles all your upstream data as one flow file (no need
> for split) and generates the SQL (no need for replace) and executes it
> for all the records in the flow file (no need for PutSQL). There are
> some reasons why PutDatabaseRecord wouldn't be the right choice; often
> it's when you need specific statements or ones that aren't currently
> supported (such as UPSERT), or when you're using a different Put than
> PutSQL (such as PutHiveQL). Otherwise I always recommend
> PutDatabaseRecord when prudent.
>
> Regards,
> Matt
>
> On Wed, Aug 15, 2018 at 10:44 AM Lone Pine Account <lpcap@roobricks.com>
> wrote:
> >
> > I have a simple flow that takes the output of a ReplaceText processor
> and sends it to PutSQL.
> >
> > This has been working in the past with a "toy" configuration.  Now that
> I'm testing it on a larger input set, it is not working but I can't figure
> out why.
> >
> > Attached is my setup.  The ReplaceText processor fills up the success
> queue, and rather than processing these it appears that the PutSQL just
> keeps filling up its "In" port with more copies of them - the number of
> "In" for PutSQL keeps going up, but nothing comes off the success queue.
> >
> > So I'm trying to debug:
> > - I've attached LogAttribute processors to the ReplaceText processor,
> and verified that the SQL commands are correct.
> > - I've attached LogAttribute processors to all of the PutSQL
> relationships, and none receive output.
> > - I've looked through the app logs and there is nothing coming up for my
> PutSQL processor.
> >
> > Where can I look to figure out why the PutSQL processor is no longer
> writing to the database?
> >
> > Thanks
>

Mime
View raw message