samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prateek Maheshwari <pmaheshw...@linkedin.com>
Subject Re: Review Request 53027: SAMZA-1017 - Added disk quota based throttling to AsyncRunLoop.
Date Wed, 19 Oct 2016 17:54:57 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53027/
-----------------------------------------------------------

(Updated Oct. 19, 2016, 10:54 a.m.)


Review request for samza and Xinyu Liu.


Changes
-------

Updated description.


Bugs: SAMZA-1017
    https://issues.apache.org/jira/browse/SAMZA-1017


Repository: samza


Description (updated)
-------

Added disk quota based throttling to AsyncRunLoop.

Overview:
Adds a Throttleable interface, implemented by RunLoop, AsyncRunLoop and ThrottlingExecutor
When AsyncRunLoop is throttled, it delays the onComplete() callback from processAsync() by
a delay amount appropriate for the desired work factor.

This implementation has a couple of known issues:
1. Adding additional delay to process()/processAsync() callback will not throttle the run
loop as long as task processing rate > message throughput. E.g., a low QPS stream with
process() time < message inter-arrival time. If desirable, this can be addressed by delaying
based on the total run loop time instead of just the process() time.

2. If throttled, users can increase their throughput back to original by increasing task.max.concurrency
and redeploying their jobs. I don't have a simple solution for this, suggestions are welcome.


Diffs
-----

  samza-core/src/main/java/org/apache/samza/container/RunLoopFactory.java a789d04 
  samza-core/src/main/java/org/apache/samza/container/disk/WatermarkDiskQuotaPolicy.java 21fbca2

  samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0 
  samza-core/src/main/java/org/apache/samza/util/Throttleable.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/util/ThrottlingExecutor.java afcc4c5 
  samza-core/src/main/java/org/apache/samza/util/ThrottlingScheduler.java PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 538ebb8 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 05a996c 
  samza-core/src/test/java/org/apache/samza/task/TestAsyncRunLoop.java ca913de 
  samza-core/src/test/java/org/apache/samza/util/TestThrottlingScheduler.java PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala aa1a8d6 
  samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cff6b96 

Diff: https://reviews.apache.org/r/53027/diff/


Testing
-------

Tested locally with a hello world app.


Thanks,

Prateek Maheshwari


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message