flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bbende <...@git.apache.org>
Subject [GitHub] flink pull request: FLINK-2740 Adding flink-connector-nifi module ...
Date Thu, 01 Oct 2015 15:50:48 GMT
Github user bbende commented on the pull request:

    https://github.com/apache/flink/pull/1198#issuecomment-144769010
  
    I think the example data flow in NiFi is producing data fairly slowly, this is based on
the scheduling tab of the GenerateFlowFile processor being set to 2 secs. We could set this
to 0 sec to start really going by stopping the processor, right-clicking and selecting Configure.

    
    The SiteToSiteClientConfig has options to set batch count but the example isn't setting
it so it is using some default, and the first instance of the source probably runs and pulls
all the available data, and when the second instance runs nothing is left.
    
    If you set the batch count lower in the example, like this:
    
        SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
    				.url("http://localhost:8080/nifi")
    				.portName("Data for Flink")
    				.requestBatchCount(5)
    				.buildConfig();
    
    and then have more than 5 flow files queued in NiFi when you start the source topology,
I think the other instances will start pulling as well. This seemed to work for me using parallelism
of 2, but of course let me know if you are seeing something different.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message