nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Setting thread counts based on worker node size
Date Mon, 25 Jan 2021 19:05:13 GMT
Hi Chris,

There’s a YouTube video [1] that I created a few months ago that discusses a lot of the
considerations to take into account when determining the number of threads to allocate to
the overall thread pool and how many to allocate to individual processors. Unfortunately,
it’s pretty non-trivial. One thing to note, though, is that the number of nodes shouldn’t
really determine the size of the thread pool, as the thread pool is per-node.

Thanks
-Mark

[1] https://www.youtube.com/watch?v=pZq0EbfDBy4


On Jan 25, 2021, at 12:05 PM, Chris Nava <chrisnava07@gmail.com<mailto:chrisnava07@gmail.com>>
wrote:

Good morning from Denver!

I'm having a hard time trying to find the answer within the documentation
online. I am fairly new to NiFi and was trying to research the possibility
of adjusting the maximum thread count of a process based on the
environment and the node size.  We have the following code and I FEEL like
doing some kind of request to see how many nodes are on the given process
and then replacing the thread_count with some equation based on the return
is the way to go. Am I on the right track or way off-base here?:

def update_thread():
   """ Updates the overall thread count for NiFi """
   if (os.getenv('NIFI_THREAD_OVERRIDE') == 'true'):
       thread_count = int(os.getenv('NIFI_THREAD_COUNT'))
   else:
       thread_count = int(round(multiprocessing.cpu_count()*0.8*4))
   api = nipyapi.nifi.apis.controller_api.ControllerApi()
   controller = api.get_controller_config()
   controller.component =
nipyapi.nifi.models.controller_configuration_dto.ControllerConfigurationDTO(thread_count)
   api.update_controller_config(controller)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message