nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: Set priority to files based on date time value stored on attribute
Date Fri, 09 Jun 2017 07:21:38 GMT
Hi Andre,

Epoch format is indeed incremental, but EnforceOrder needs every
number without skip.
Unless the use-case guarantees incoming FlowFiles has a perfect set of
incremental numbers in successful flow, EnforceOrder is not helpful.
E.g. 1, 5, 4, 2, 3 can be enforced by EnforceOrder
but 100, 500, 400, 200, 300 can NOT. But PriorityAttributePrioritizer
can sort these FlowFiles.

However, a caveat is, PriorityAttributePrioritizer does not wait if
some incoming FlowFiles are delayed.
If 1, 5 are queued when a processor is scheduled,
PriorityAttributePrioritizer picks 1 and 5. It does not care missing
numbers.

On the other hand, EnforceOrder does wait.
If 1, 5 are queued, EnforceOrder transfers 1, then keep 5 wait.
Then if 4, 2, 3 arrive, EnforceOrder transfers 2, 3, 4, then 5

Thanks,
Koji

On Thu, Jun 8, 2017 at 9:28 PM, Manojkumar Ravichandran
<sendmailtomj@gmail.com> wrote:
> Hi Pierre,
>
>
> After converting those date time format in to integer (using expression
> language),I can able to process the file as per the requirement
> by setting those integer values to the priority attribute and process those
> files based on that priority.
>
> Thanks for your guidance
>
>
> Regards,
> Manoj kumar R
>
>
> On Thu, Jun 8, 2017 at 8:50 AM, Pierre Villard <pierre.villard.fr@gmail.com>
> wrote:
>>
>> Hi Manoj,
>>
>> You may want ot have a look at EnforceOrder processor [1] or simply the
>> prioritizers [2] of the connections (it depends of how your workflow is
>> working). The idea would be to extract the date as an attribute of your flow
>> file, convert into an integer (using expression language) and use it to
>> ensure order.
>>
>> [1]
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
>> [2]
>> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization
>>
>> Hope this helps.
>>
>>
>> 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran
>> <sendmailtomj@gmail.com>:
>>>
>>> Hi All,
>>>
>>> I need to process the files based on the date time value stored on the
>>> attribute
>>>
>>> For example:
>>>
>>> If the incoming files contains the following date time attribute values
>>>
>>> 2017/06/07  16:57:02
>>> 2017/06/06  12:49:49
>>> 2017/06/06  11:09:28
>>> 2017/06/06  06:37:45
>>>
>>> I need to process the files based on the order of time that is oldest one
>>> from the current time
>>>
>>> First I want to access the file that contains below date time attribute
>>> which is the oldest one among them from the current time
>>> i.e 2017/06/06  06:37:45
>>> and then below one,
>>> 2017/06/06  11:09:28
>>> and then this
>>> 2017/06/06  12:49:49
>>> so on ....
>>>
>>> How can I achieve the above mentioned scenario ?
>>>
>>> Regards,
>>> Manoj kumar R
>>
>>
>

Mime
View raw message