samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject Re: How do you monitor Samza?
Date Mon, 12 Sep 2016 21:43:14 GMT
Thanks for sharing Nicolas, This is super-useful.

On Mon, Sep 12, 2016 at 10:56 AM, Nicolas Maquet <nmaquet@gmail.com> wrote:

> Hey Tommy,
>
> At Movio, we feed all of our Samza metrics into Prometheus and use Grafana
> to create visual dashboards. We configure all of our jobs to produce
> metrics into a single topic and we have a monitoring tool that consumes
> that topic and exposes the data to Prometheus (
> https://github.com/movio/samza-prometheus-exporter).
>
> We've been using this setup in production for 6 months now and it's been
> working really well for us.
>
> Nick
>
>
> On 13 Sep 2016 1:04 a.m., "Tommy Becker" <tobecker@tivo.com> wrote:
>
> > One of the challenges we've faced with deploying Samza is effectively
> > monitoring it. We currently use the MetricsSnapshotReporter and consume
> the
> > resulting stream pulling out specific values that we want to monitor, but
> > this approach isn't scalable. We're exploring pumping the metrics into an
> > ELK stack, but the format isn't ideal (many keys contain dots, which ES
> > doesn't like). Additionally, many of the keys are "dynamic" in that the
> > names are determined by runtime configuration, which makes pulling
> specific
> > values out a bit of a pain. I'd love to get some information on how other
> > folks are monitoring their Samza jobs.
> >
> > --
> > Tommy Becker
> > Senior Software Engineer
> >
> > Digitalsmiths
> > A TiVo Company
> >
> > www.digitalsmiths.com<http://www.digitalsmiths.com>
> > tobecker@tivo.com<mailto:tobecker@tivo.com>
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message