samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luis Fernando De Pombo <lfdepo...@gmail.com>
Subject Re: what does the utilization metric in SamzaContainerMetrics show?
Date Tue, 07 Jul 2015 19:12:35 GMT
Shadi, you are absolutely right. I created a diff (
https://reviews.apache.org/r/36274/) to properly calculate activeMs. Feel
free to leave any comments there. Thanks for pointing this out :)
ᐧ

On Fri, Jun 26, 2015 at 9:40 AM, Shadi Noghabi <
snoghabi@linkedin.com.invalid> wrote:

>
> Thanks for your prompt reply. I just don’t see when will there be an idle
> time that is not added to the activeMs but is counted in the totalMs.
> Because as far as I can tell, if one of the process,window, or commit
> takes more time it will be reflected in the activeMs value as well.
>
>
>
> On 6/25/15, 7:47 PM, "Luis Fernando De Pombo" <lfdepombo@gmail.com> wrote:
>
> >Hi Shadi,
> >Thanks for asking. This metric tracks the utilization of the event loop
> ><
> https://samza.apache.org/learn/documentation/0.8/container/event-loop.htm
> >l>
> >within a samza container, which uses a single thread, that is in charge of
> >reading and writing messages, flushing metrics, checkpointing, and
> >windowing. It is important to track the utilization (aka "duty cycle") of
> >any event loop, which is the sum of all the timings (activeMs) divided by
> >the window length (totalMs).
> >
> >You are right in that most of the time this value will be close to 1,
> >which
> >represents complete utilization. However when the event loop starts to
> >have
> >idle time, this metric will give you an idea of how much headroom you have
> >before the job will start to seriously fall behind.
> >
> >I hope that answers your question!
> >
> >ᐧ
> >
> >On Thu, Jun 25, 2015 at 6:39 PM, Shadi Noghabi <
> >snoghabi@linkedin.com.invalid> wrote:
> >
> >> Hi,
> >>
> >> I was wondering what does this utilization metric in the
> >> SamzaContainerMetrics show? I am asking this sine in the code it is
> >> calculated as below:
> >>
> >> while (!shutdownNow) {
> >>   val loopStartTime = clock();
> >>   process
> >>   window
> >>   commit
> >>   val totalMs = clock() - loopStartTime
> >>   metrics.utilization.set(activeMs.toFloat/totalMs)
> >>   activeMs = 0L
> >> }
> >>
> >> Where the totalMs is the time it takes to start calling process until
> >> commit is done which is equal to  the time it takes to run process,
> >>window,
> >> and commit. And they way activeMs is calculated is by summing up the
> >>time
> >> it takes to call process, window and commit, which means these two
> >>values
> >> are going to be almost the same and the utilization is always going to
> >>be
> >> almost 1.
> >>
> >> I was just wondering what is the point of doing this?
> >>
> >>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message