samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prateek Maheshwari <>
Subject Re: Review Request 53027: SAMZA-1017 - Added disk quota based throttling to AsyncRunLoop.
Date Wed, 19 Oct 2016 17:54:57 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Oct. 19, 2016, 10:54 a.m.)

Review request for samza and Xinyu Liu.


Updated description.

Bugs: SAMZA-1017

Repository: samza

Description (updated)

Added disk quota based throttling to AsyncRunLoop.

Adds a Throttleable interface, implemented by RunLoop, AsyncRunLoop and ThrottlingExecutor
When AsyncRunLoop is throttled, it delays the onComplete() callback from processAsync() by
a delay amount appropriate for the desired work factor.

This implementation has a couple of known issues:
1. Adding additional delay to process()/processAsync() callback will not throttle the run
loop as long as task processing rate > message throughput. E.g., a low QPS stream with
process() time < message inter-arrival time. If desirable, this can be addressed by delaying
based on the total run loop time instead of just the process() time.

2. If throttled, users can increase their throughput back to original by increasing task.max.concurrency
and redeploying their jobs. I don't have a simple solution for this, suggestions are welcome.


  samza-core/src/main/java/org/apache/samza/container/ a789d04 
  samza-core/src/main/java/org/apache/samza/container/disk/ 21fbca2

  samza-core/src/main/java/org/apache/samza/task/ a510bb0 
  samza-core/src/main/java/org/apache/samza/util/ PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/util/ afcc4c5 
  samza-core/src/main/java/org/apache/samza/util/ PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 538ebb8 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 05a996c 
  samza-core/src/test/java/org/apache/samza/task/ ca913de 
  samza-core/src/test/java/org/apache/samza/util/ PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala aa1a8d6 
  samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cff6b96 



Tested locally with a hello world app.


Prateek Maheshwari

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message