nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Dean <>
Subject Efficiently caching API results in a NiFi controller service
Date Tue, 01 May 2018 04:01:49 GMT

I have a custom NiFi controller service that retrieves data from an external web service via
HTTP requests. The results from these HTTP requests will be needed at various points throughout
my process flow. In some situations, I could end up needing to access the HTTP response dozens
or even hundreds of times. 

Given that the results of the HTTP request rarely change, I’d like them to be cached by
my service and returned to my processors when needed. I’d need some way to explicitly clear
the cache for those occasions when the data in the service does change.

I’ve looked at using the DistributedMapCacheClientService implementation to cache my web
service’s results, but it seems like that connects to a server via a socket connection and
that doesn’t seem like it would be all that much more efficient than calling the web service
directly. I’ve also looked at using the service’s state manager to store the results as
state, but my data is a little more complex than what the documentation for state suggests
is optimal: I don’t think my total map size will get to 1MB in size but it could be possible.

Am I overthinking this? Would a simpler solution like creating a simple Java HashMap inside
my controller service be adequate? I could empty the contents of the hash map whenever the
controller services is enabled/disabled. Would the memory used by this kind of simplified
local caching cause problems somewhere down the line?

Are there other caching strategies I should be considering?



View raw message