kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachin Mittal <sjmit...@gmail.com>
Subject Re: Need some help in identifying some important metrics to monitor for streams
Date Sun, 05 Mar 2017 07:14:29 GMT
Yes setting client id works. Now we are able to add metrics as part of our
cron job.

One additional question I have is
http://kafka.apache.org/documentation.html#kafka_streams_monitoring

I am monitoring the commit latency and process latency.
Commit latency is usually say in 1000ms and process latency is usually say
in 1ms.
So 3 order of magnitude less than commit latency.

This makes sense because in our commit phase ie forEach we do some external
db operation.

I just wanted to understand say in single poll request if it fetches n
records does the above values indicate time computed for all n records or
just a single record.

or is it the total average time to process these records = n * process
latency + commit latency  before making another poll request.

Basically we just want to know how often is poll getting called just to see
how close is it to MAX_POLL_INTERVAL_MS_CONFIG.

Thanks
Sachin


On Sun, Mar 5, 2017 at 11:42 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> That is right, since client-id is used as the metrics name which should be
> distinguishable.
>
> https://kafka.apache.org/documentation/#streamsconfigs (I think we can
> improve on the explanation of the client.id config)
>
> A common client-id could contain the machine's host-port; of course, if you
> have more than one Streams instances running on the same machine that wont
> work and you need to consider using more information.
>
> Again the client-id config is not required, and when not specified Streams
> will use an UUID suffix to achieve uniqueness but as you observed it is
> less human readable for monitoring.
>
>
> Guozhang
>
> On Fri, Mar 3, 2017 at 5:18 PM, Sachin Mittal <sjmittal@gmail.com> wrote:
>
> > Son if I am running my stream and across a cluster of different machine
> > each machine should have a different client id.
> >
> > On 4 Mar 2017 12:36 a.m., "Guozhang Wang" <wangguoz@gmail.com> wrote:
> >
> > > Sachin,
> > >
> > > The reason that you got metrics name as
> > >
> > > new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-StreamThread-1
> > >
> > >
> > > Is that you did not set the "CLIENT_ID_CONFIG" in your app, and
> > > KafkaStreams have to use a default combo of "appID:
> > > new-part-advice"-"processID: a UUID to guarantee uniqueness across
> > > machines" as its clientId.
> > >
> > >
> > > As for metricsName, it is always set as "clientId + "-" + threadName"
> > where
> > > "StreamThread-1" is your threadName which is unique WITHIN the JVM and
> > that
> > > is why we still need the globally unique clientId for distinguishment.
> > >
> > > I just checked the source code and this logic was not changed from
> 0.10.1
> > > to 0.10.2, so I guess you set your clientId as "new-advice-1" as well
> in
> > > 0.10.1?
> > >
> > >
> > > Guozhang
> > >
> > >
> > >
> > > On Fri, Mar 3, 2017 at 4:02 AM, Eno Thereska <eno.thereska@gmail.com>
> > > wrote:
> > >
> > > > Hi Sachin,
> > > >
> > > > Now that the confluent platform 3.2 is out, we also have some more
> > > > documentation on this here: http://docs.confluent.io/3.2.
> > > > 0/streams/monitoring.html <http://docs.confluent.io/3.2.
> > > > 0/streams/monitoring.html>. We added a note on how to add other
> > metrics.
> > > >
> > > > Yeah, your calculation on poll time makes sense. The important
> metrics
> > > are
> > > > the “info” ones that are on by default. However, for stageful
> > > applications,
> > > > if you suspect that state stores might be bottlenecking, you might
> want
> > > to
> > > > collect those metrics too.
> > > >
> > > > On the benchmarks, the one called “processstreamwithstatestore” and
> > > > “count” are the closest to a benchmarking on RocksDb with the default
> > > > configs. The first writes each record to RocksDb, while the second
> > > performs
> > > > simple aggregates (reads and writes from/to RocksDb).
> > > >
> > > > We might need to add more benchmarks here, would be great to get some
> > > > ideas and help from the community. E.g., a pure RocksDb benchmark
> that
> > > > doesn’t go through streams at all.
> > > >
> > > > Could you open a JIRA on the name issue please? As an “improvement”.
> > > >
> > > > Thanks
> > > > Eno
> > > >
> > > >
> > > >
> > > > > On Mar 2, 2017, at 6:00 PM, Sachin Mittal <sjmittal@gmail.com>
> > wrote:
> > > > >
> > > > > Hi,
> > > > > I had checked the monitoring docs, but could not figure out which
> > > metrics
> > > > > are important ones.
> > > > >
> > > > > Also mainly I am looking at the average time spent between 2
> > successive
> > > > > poll requests.
> > > > > Can I say that average time between 2 poll requests is sum of
> > > > >
> > > > > commit + poll + process + punctuate (latency-avg).
> > > > >
> > > > >
> > > > > Also I checked the benchmark tests results but could not find any
> > > > > information on rocksdb metrics for fetch and put operations.
> > > > > Is there any benchmark for these or based on my values in previous
> > mail
> > > > can
> > > > > something be commented on its performance.
> > > > >
> > > > >
> > > > > Lastly can we get some help on names like
> > > new-part-advice-d1094e71-0f59-
> > > > > 45e8-98f4-477f9444aa91-StreamThread-1 and have more standard name
> of
> > > > thread
> > > > > like new-advice-1-StreamThread-1(as in version 10.1.1) so we can
> log
> > > > these
> > > > > metrics as part of out cron jobs.
> > > > >
> > > > > Thanks
> > > > > Sachin
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Mar 2, 2017 at 9:31 PM, Eno Thereska <
> eno.thereska@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > >> Hi Sachin,
> > > > >>
> > > > >> The new streams metrics are now documented at
> > > https://kafka.apache.org/
> > > > >> documentation/#kafka_streams_monitoring <
> https://kafka.apache.org/
> > > > >> documentation/#kafka_streams_monitoring>. Note that not all
of
> them
> > > are
> > > > >> turned on by default.
> > > > >>
> > > > >> We have several benchmarks that run nightly to monitor streams
> > > > >> performance. They all stem from the SimpleBenchmark.java
> benchmark.
> > In
> > > > >> addition, their results are published nightly here
> > > > >> http://testing.confluent.io <http://testing.confluent.io/>,
> (e.g.,
> > > > under
> > > > >> the trunk results). E.g., looking at today's results:
> > > > >> http://confluent-kafka-system-test-results.s3-us-west-2.
> > > > >> amazonaws.com/2017-03-02--001.1488449554--apache--trunk--
> > > > >> ef92bb4/report.html <http://confluent-kafka-
> > > system-test-results.s3-us-
> > > > >> west-2.amazonaws.com/2017-03-02--001.1488449554--apache--
> > > > >> trunk--ef92bb4/report.html>
> > > > >> (if you search for "benchmarks.streams") you'll see results from
a
> > > > series
> > > > >> of benchmarks, ranging from simply consuming, to simple topologies
> > > with
> > > > a
> > > > >> source and sink, to joins and count aggregate. These run on AWS
> > > nightly,
> > > > >> but you can also run manually on your setup.
> > > > >>
> > > > >> In addition, programmatically the code can check the
> > > > KafkaStreams.state()
> > > > >> and register listeners for when the state changes. For example,
> the
> > > > state
> > > > >> can change from "running" to "rebalancing".
> > > > >>
> > > > >> It is likely we'll need more metrics moving forward and would
be
> > great
> > > > to
> > > > >> get feedback from the community.
> > > > >>
> > > > >>
> > > > >> Thanks
> > > > >> Eno
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>> On 2 Mar 2017, at 11:54, Sachin Mittal <sjmittal@gmail.com>
> wrote:
> > > > >>>
> > > > >>> Hello All,
> > > > >>> I had few questions regarding monitoring of kafka streams
> > application
> > > > and
> > > > >>> what are some important metrics we should collect in our
case.
> > > > >>>
> > > > >>> Just a brief overview, we have a single thread application
> > (0.10.1.1)
> > > > >>> reading from single partition topic and it is working all
fine.
> > > > >>> Then we have same application (using 0.10.2.0) multi threaded
> with
> > 4
> > > > >>> threads per machine and 3 machines cluster setup reading
for same
> > but
> > > > >>> partitioned topic (12 partitions).
> > > > >>> Thus we have each thread processing single partition same
case as
> > > > earlier
> > > > >>> one.
> > > > >>>
> > > > >>> The new setup also works fine in steady state, but under
load
> > somehow
> > > > it
> > > > >>> triggers frequent re-balance and then we run into all sort
of
> > issues
> > > > like
> > > > >>> stream thread dying due to CommitFailedException or entering
into
> > > > >> deadlock
> > > > >>> state.
> > > > >>> After a while we restart all the instances then it works
fine
> for a
> > > > while
> > > > >>> and again we get the same problem and it goes on.
> > > > >>>
> > > > >>> 1. So just to monitor, like when first thread fails what
would be
> > > some
> > > > >>> important metrics we should be collecting to get some sense
of
> > whats
> > > > >> going
> > > > >>> on?
> > > > >>>
> > > > >>> 2. Is there any metric that tells time elapsed between successive
> > > poll
> > > > >>> requests, so we can monitor that?
> > > > >>>
> > > > >>> Also I did monitor rocksdb put and fetch times for these
2
> > instances
> > > > and
> > > > >>> here is the output I get:
> > > > >>> 0.10.1.1
> > > > >>> $>get -s  -b kafka.streams:type=stream-
> > > rocksdb-window-metrics,client-
> > > > >> id=new-advice-1-StreamThread-1
> > > > >>> key-table-put-avg-latency-ms
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-advice-1-StreamThread-1:
> > > > >>> 206431.7497615029
> > > > >>> $>get -s  -b kafka.streams:type=stream-
> > > rocksdb-window-metrics,client-
> > > > >> id=new-advice-1-StreamThread-1
> > > > >>> key-table-fetch-avg-latency-ms
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-advice-1-StreamThread-1:
> > > > >>> 2595394.2746129474
> > > > >>> $>get -s  -b kafka.streams:type=stream-
> > > rocksdb-window-metrics,client-
> > > > >> id=new-advice-1-StreamThread-1
> > > > >>> key-table-put-qps
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-advice-1-StreamThread-1:
> > > > >>> 232.86299499317252
> > > > >>> $>get -s  -b kafka.streams:type=stream-
> > > rocksdb-window-metrics,client-
> > > > >> id=new-advice-1-StreamThread-1
> > > > >>> key-table-fetch-qps
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-advice-1-StreamThread-1:
> > > > >>> 373.61071016166284
> > > > >>>
> > > > >>> Same values for 0.10.2.0 I get
> > > > >>> $>get -s -b kafka.streams:type=stream-
> > rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > StreamThread-1
> > > > >>> key-table-put-latency-avg
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > > StreamThread-1:
> > > > >>> 1199859.5535022356
> > > > >>> $>get -s -b kafka.streams:type=stream-
> > rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > StreamThread-1
> > > > >>> key-table-fetch-latency-avg
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > > StreamThread-1:
> > > > >>> 3679340.80748852
> > > > >>> $>get -s -b kafka.streams:type=stream-
> > rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > StreamThread-1
> > > > >>> key-table-put-rate
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > > StreamThread-1:
> > > > >>> 56.134778706069184
> > > > >>> $>get -s -b kafka.streams:type=stream-
> > rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > StreamThread-1
> > > > >>> key-table-fetch-rate
> > > > >>> #mbean = kafka.streams:type=stream-
> rocksdb-window-metrics,client-
> > > > >>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
> > > > StreamThread-1:
> > > > >>> 136.10721427931827
> > > > >>>
> > > > >>> I notice that result in 10.2.0 is much worse than same for
10.1.1
> > > > >>>
> > > > >>> I would like to know
> > > > >>> 1. Is there any benchmark on rocksdb as at what rate/latency
it
> > > should
> > > > be
> > > > >>> doing put/fetch operations.
> > > > >>>
> > > > >>> 2. What could be the cause of inferior numbers in 10.2.0,
is it
> > > because
> > > > >>> this application is also running three other threads doing
the
> same
> > > > >> thing.
> > > > >>>
> > > > >>> 3. Also whats with the name new-part-advice-d1094e71-
> > > > >>> 0f59-45e8-98f4-477f9444aa91-StreamThread-1
> > > > >>>   I wanted to put this as a part of my cronjob, so why can't
we
> > have
> > > > >>> simpler name like we have in 10.1.1, so it is easy to write
the
> > > script.
> > > > >>>
> > > > >>> Thanks
> > > > >>> Sachin
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message