nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JPercivall <...@git.apache.org>
Subject [GitHub] nifi issue #349: NIFI-1767 AWS IoT processors
Date Wed, 01 Jun 2016 16:08:42 GMT
Github user JPercivall commented on the issue:

    https://github.com/apache/nifi/pull/349
  
    I actually already committed general MQTT processors [1] and they have some good "best
practices" with regards to publishing/consuming messages and MQTT I think would be good to
follow here. The first few I have are below:
    
    First in regards to the internal queue of messages, there needs to be some kind of limit
on it. If a user doesn't set a fast enough trigger duration the queue could become enormous
and, since all the messages are held in memory, potentially bringing NiFi to a halt. Here
is the property I add to ConsumeMQTT[2].
    
    This leads to the second, what to do with the internal queue when the processor is stopped.
There will potentially be some data left in the internal queue when the processor is stopped
and if that data is not handled it will more than likely be lost. In the OnStopped I actually
transfer all messages in the queue (just like is done int he OnTrigger)[3].
    
    In the PutAWSIoT processor, you read the attributes from the FlowFile to essentially overwrite
Processor defined properties. Being able to have different QoS/Topics/Retained different messages
is very important and the most user friendly way to do this is through Expression Language.
Check out how I do it here[4]. This gives the user to potentially dynamically select the QoS/Topic/Retained
and doesn't do things automatically for the user. 
    
    [1] https://github.com/apache/nifi/tree/f47af1ce8336c9305916f00738976f3505b01b0b/nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt
    [2] https://github.com/apache/nifi/blob/f47af1ce8336c9305916f00738976f3505b01b0b/nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java#L111
    [3] https://github.com/apache/nifi/blob/f47af1ce8336c9305916f00738976f3505b01b0b/nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java#L233
    [4] https://github.com/apache/nifi/blob/f47af1ce8336c9305916f00738976f3505b01b0b/nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/PublishMQTT.java#L63


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message