samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Shkolnik <bor...@apache.org>
Subject Re: Review Request 44604: split deployment logic
Date Thu, 31 Mar 2016 21:12:59 GMT


> On March 25, 2016, 5:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala, line
17
> > <https://reviews.apache.org/r/44604/diff/2/?file=1299997#file1299997line17>
> >
> >     One question on this: how about ThreadJobFactory? Is it the current limitation
that split deployment only works for ProcessJobFactory and YarnJobFactory? Is it possible
to set fwkPath for ThreadJobFactory as well? If yes, I would recommend to open a JIRA to track
that, and document it.

ThreadJobFactory is the only one that doesn't need it, because it is using the same phisical
process of the JobRunner, which was run from the correct location (of the split deployment).
ProcessJobFactory, OTOH, needs to build the path to the executable, and thus needs the setting.


> On March 25, 2016, 5:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java, line 109
> > <https://reviews.apache.org/r/44604/diff/2/?file=1300001#file1300001line109>
> >
> >     This segment of code is almost dup from the ProcessJobFactory. Can we fold this
in the config.get()? Or some utility class?

created a common method.


- Boris


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44604/#review125429
-----------------------------------------------------------


On March 15, 2016, 5:33 p.m., Boris Shkolnik wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44604/
> -----------------------------------------------------------
> 
> (Updated March 15, 2016, 5:33 p.m.)
> 
> 
> Review request for samza and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-849
>     https://issues.apache.org/jira/browse/SAMZA-849
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> samza split deployment. See SAMZA-849 for details and design.
> 
> 
> Diffs
> -----
> 
>   samza-api/src/main/java/org/apache/samza/job/CommandBuilder.java 97bb05afe7ccd17a9a2443cd1cdf96dd8d1cc740

>   samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 4f3e9a297ce2c0df0f5f25e0aad62f7bed774cd6

>   samza-core/src/main/scala/org/apache/samza/job/ShellCommandBuilder.scala c3f60acf9ca88326ef7f2cfbbff2289f70cdd46e

>   samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 0792a59cb7b042220c8dbfc0713c5ef42e93ab25

>   samza-core/src/test/scala/org/apache/samza/job/TestShellCommandBuilder.scala fd014b326c8cca914513fa3ac9d913ea78deca23

>   samza-shell/src/main/bash/run-class.sh 46603c50f1fe02758aae5a568f06b1a65dff9734 
>   samza-shell/src/main/bash/run-container.sh 069279c88c88bd7d8b4a37510b38094446bb034b

>   samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java 6580b9ab2cb293c98f31fee2766db7471e5d99ee

>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 1aa26bba00d2c3d42ef3b8951fa723ffff14d410

>   samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaTaskManager.java 9da1edf6ff165ef0306de8730853ad30551a9831

> 
> Diff: https://reviews.apache.org/r/44604/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message