flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bbende <...@git.apache.org>
Subject [GitHub] flink pull request: FLINK-2740 Adding flink-connector-nifi module ...
Date Thu, 01 Oct 2015 16:00:27 GMT
Github user bbende commented on the pull request:

    https://github.com/apache/flink/pull/1198#issuecomment-144771981
  
    I realized I didn't fully answer your question about how that works in a cluster... the
SiteToSiteClient knows about all the nodes in the NiFi cluster and will pull from the output
port on each node. Same concept on the sending side, it will distribute the data to different
nodes in the cluster, sending more data to nodes that are considered less busy. The SiteToSiteClient
is what we use internally for two NiFi instances/clusters to communicate with each other.

    
    Regarding how they know they belong to the same system, technically they don't, and other
non-Flink clients could pull from that same port, but we have to hope that if an organization
is using Flink and NiFi they would be ensuring that port was only being used by this specific
Flink streaming process, and there could be multiple output ports to support multiple Flink
streaming processes pulling.
    
    There is also the option to secure a NiFi instance with SSL, and then provide SSL credentials
to the SiteToSite client in order lockdown NiFi so that only authorized clients can pull/push
data.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message