samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navina Ramesh" <nram...@linkedin.com>
Subject Re: Review Request 41663: SAMZA-843 : Slow start of Samza jobs with large number of containers
Date Mon, 04 Jan 2016 23:20:43 GMT


> On Jan. 4, 2016, 11:11 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala, line
204
> > <https://reviews.apache.org/r/41663/diff/2-3/?file=1181044#file1181044line204>
> >
> >     I am a bit confused here. If the refreshJobModel() function actually returns
a different jobModel than what's already set in jobModelRef, isn't the whole point is to replace
it w/ the new one?
> 
> Yi Pan (Data Infrastructure) wrote:
>     Ah, I saw it now. Actually, I think that we should keep this line in initializaJobModel()
method, but remove the following lines in getCoordinator():
>     {code}
>         val jobModel: JobModel = jobModelGenerator()
>         jobModelRef.set(jobModel)
>     {code}

The thing I noticed is that refreshJobModel and initializeJobModel are both private. The only
time when the job model is getting refreshed is when getJobCoordinator is called. Now that
I think about it, after the change to the JobServlet, we don't really make a call to refresh
the job model. The only other place where we do a jobCoordinator.jobmodel() is in SamzaTaskManager.
AT this point, we use the jobModel simply to invoke the readContainerLocality method on it.
Technically, nothing refreshes the jobModel after initialization. Better discussed in person
:)


- Navina


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41663/#review112677
-----------------------------------------------------------


On Jan. 4, 2016, 10:40 p.m., Navina Ramesh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41663/
> -----------------------------------------------------------
> 
> (Updated Jan. 4, 2016, 10:40 p.m.)
> 
> 
> Review request for samza, Boris Shkolnik, Yan Fang, Chris Riccomini, Jake Maes, Jagadish
Venkatraman, Xinyu Liu, and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-843
>     https://issues.apache.org/jira/browse/SAMZA-843
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> SAMZA-843 : Slow start of Samza jobs with large number of containers
> 
> 
> Diffs
> -----
> 
>   samza-autoscaling/src/main/java/org/apache/samza/autoscaling/deployer/ConfigManager.java
87346bc9f3d92e3ae86ed264c82d12a4d27f3188 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala ddce1481cee41dfb997103dbe3d5df44a123564e

>   samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 112ec1c2935fb6c2ebe9b5d9090263cea94c0cf9

>   samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala a3baddbe81c96f58e718e062e964485eaa2dc701

>   samza-core/src/main/scala/org/apache/samza/util/Util.scala 58fbb8f8177a109d35659a29ef6660e239334de2

>   samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 365ff0a8ece808a8143ee3580f1f42238dd292d1

>   samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala 80cccf3104e6e0ce7457303f40baa2f4fa807782

> 
> Diff: https://reviews.apache.org/r/41663/diff/
> 
> 
> Testing
> -------
> 
> Added some additional unit tests
> ./gradlew clean build
> 
> 
> Thanks,
> 
> Navina Ramesh
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message