nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject MergeRecord issue - Could not merge bin .. encountered before this bin was evicted
Date Fri, 03 Apr 2020 13:27:59 GMT
Hey guys,

weird issue, NiFi 1.9.2 both clustered mode and a single NiFi node.

I have this process below that runs every 15 minutes. 4 or 5 times a day a
get an error below now in the past few days while it worked fine for weeks
with no errors at all. Help me understand why please.

from help file:

*If the <Merge Strategy> property is set to "Defragment" then a bin is full
only when the number of FlowFiles in the bin is equal to the number
specified by the "fragment.count" attribute of one of the FlowFiles in the
bin*. All FlowFiles that have this attribute must have the same value for
this attribute, or else they will be routed to the "failure" relationship.
It is not necessary that all FlowFiles have this value, but at least one
FlowFile in the bin must have this value or the bin will never be complete.
If all of the necessary FlowFiles are not binned together by the point at
which the bin times amount (*as specified by the <Max Bin Age> property),*
then the FlowFiles will all be routed to the 'failure' relationship instead
of being merged together.


I never set MAx Bin Age or Max size and till now it worked fine. Was this
behaviour changed possibly and now either max age or max size needs to be
set for defragement strategy?

[image: image.png]
bulletin error

[{"objectId":"b297e4a7-cc7d-4e66-87bc-3cc46a72d0a4","platform":"nifi","bulletinId":518,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.252Z"},{"objectId":"c21832e8-27bf-4f5b-a373-06c94256a08a","platform":"nifi","bulletinId":517,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.244Z"},{"objectId":"4cbc4dd5-3326-49db-ad62-2898282f285e","platform":"nifi","bulletinId":516,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.243Z"},{"objectId":"6c039c2c-3640-4890-8afd-dfdb74aafcf3","platform":"nifi","bulletinId":515,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.241Z"},{"objectId":"1fd7f1f4-c45e-4b31-a5b7-71ce5014128d","platform":"nifi","bulletinId":514,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.240Z"}]

Mime
View raw message