kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eno Thereska <eno.there...@gmail.com>
Subject Re: Need some help in identifying some important metrics to monitor for streams
Date Sun, 05 Mar 2017 08:43:39 GMT
Answer inline:

> 
> I just wanted to understand say in single poll request if it fetches n
> records does the above values indicate time computed for all n records or
> just a single record.
> 

In 0.10.2, the process latency is that of a single record, not the sum of n records. The commit
latency is the latency for several requests. So your second statement is true:

> or is it the total average time to process these records = n * process
> latency + commit latency  before making another poll request.

Correct.

Thanks
Eno



> 
> Basically we just want to know how often is poll getting called just to see
> how close is it to MAX_POLL_INTERVAL_MS_CONFIG.
> 
> Thanks
> Sachin
> 
> 
> On Sun, Mar 5, 2017 at 11:42 AM, Guozhang Wang <wangguoz@gmail.com> wrote:
> 
>> That is right, since client-id is used as the metrics name which should be
>> distinguishable.
>> 
>> https://kafka.apache.org/documentation/#streamsconfigs (I think we can
>> improve on the explanation of the client.id config)
>> 
>> A common client-id could contain the machine's host-port; of course, if you
>> have more than one Streams instances running on the same machine that wont
>> work and you need to consider using more information.
>> 
>> Again the client-id config is not required, and when not specified Streams
>> will use an UUID suffix to achieve uniqueness but as you observed it is
>> less human readable for monitoring.
>> 
>> 
>> Guozhang
>> 
>> On Fri, Mar 3, 2017 at 5:18 PM, Sachin Mittal <sjmittal@gmail.com> wrote:
>> 
>>> Son if I am running my stream and across a cluster of different machine
>>> each machine should have a different client id.
>>> 
>>> On 4 Mar 2017 12:36 a.m., "Guozhang Wang" <wangguoz@gmail.com> wrote:
>>> 
>>>> Sachin,
>>>> 
>>>> The reason that you got metrics name as
>>>> 
>>>> new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-StreamThread-1
>>>> 
>>>> 
>>>> Is that you did not set the "CLIENT_ID_CONFIG" in your app, and
>>>> KafkaStreams have to use a default combo of "appID:
>>>> new-part-advice"-"processID: a UUID to guarantee uniqueness across
>>>> machines" as its clientId.
>>>> 
>>>> 
>>>> As for metricsName, it is always set as "clientId + "-" + threadName"
>>> where
>>>> "StreamThread-1" is your threadName which is unique WITHIN the JVM and
>>> that
>>>> is why we still need the globally unique clientId for distinguishment.
>>>> 
>>>> I just checked the source code and this logic was not changed from
>> 0.10.1
>>>> to 0.10.2, so I guess you set your clientId as "new-advice-1" as well
>> in
>>>> 0.10.1?
>>>> 
>>>> 
>>>> Guozhang
>>>> 
>>>> 
>>>> 
>>>> On Fri, Mar 3, 2017 at 4:02 AM, Eno Thereska <eno.thereska@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi Sachin,
>>>>> 
>>>>> Now that the confluent platform 3.2 is out, we also have some more
>>>>> documentation on this here: http://docs.confluent.io/3.2.
>>>>> 0/streams/monitoring.html <http://docs.confluent.io/3.2.
>>>>> 0/streams/monitoring.html>. We added a note on how to add other
>>> metrics.
>>>>> 
>>>>> Yeah, your calculation on poll time makes sense. The important
>> metrics
>>>> are
>>>>> the “info” ones that are on by default. However, for stageful
>>>> applications,
>>>>> if you suspect that state stores might be bottlenecking, you might
>> want
>>>> to
>>>>> collect those metrics too.
>>>>> 
>>>>> On the benchmarks, the one called “processstreamwithstatestore” and
>>>>> “count” are the closest to a benchmarking on RocksDb with the default
>>>>> configs. The first writes each record to RocksDb, while the second
>>>> performs
>>>>> simple aggregates (reads and writes from/to RocksDb).
>>>>> 
>>>>> We might need to add more benchmarks here, would be great to get some
>>>>> ideas and help from the community. E.g., a pure RocksDb benchmark
>> that
>>>>> doesn’t go through streams at all.
>>>>> 
>>>>> Could you open a JIRA on the name issue please? As an “improvement”.
>>>>> 
>>>>> Thanks
>>>>> Eno
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mar 2, 2017, at 6:00 PM, Sachin Mittal <sjmittal@gmail.com>
>>> wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> I had checked the monitoring docs, but could not figure out which
>>>> metrics
>>>>>> are important ones.
>>>>>> 
>>>>>> Also mainly I am looking at the average time spent between 2
>>> successive
>>>>>> poll requests.
>>>>>> Can I say that average time between 2 poll requests is sum of
>>>>>> 
>>>>>> commit + poll + process + punctuate (latency-avg).
>>>>>> 
>>>>>> 
>>>>>> Also I checked the benchmark tests results but could not find any
>>>>>> information on rocksdb metrics for fetch and put operations.
>>>>>> Is there any benchmark for these or based on my values in previous
>>> mail
>>>>> can
>>>>>> something be commented on its performance.
>>>>>> 
>>>>>> 
>>>>>> Lastly can we get some help on names like
>>>> new-part-advice-d1094e71-0f59-
>>>>>> 45e8-98f4-477f9444aa91-StreamThread-1 and have more standard name
>> of
>>>>> thread
>>>>>> like new-advice-1-StreamThread-1(as in version 10.1.1) so we can
>> log
>>>>> these
>>>>>> metrics as part of out cron jobs.
>>>>>> 
>>>>>> Thanks
>>>>>> Sachin
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, Mar 2, 2017 at 9:31 PM, Eno Thereska <
>> eno.thereska@gmail.com
>>>> 
>>>>> wrote:
>>>>>> 
>>>>>>> Hi Sachin,
>>>>>>> 
>>>>>>> The new streams metrics are now documented at
>>>> https://kafka.apache.org/
>>>>>>> documentation/#kafka_streams_monitoring <
>> https://kafka.apache.org/
>>>>>>> documentation/#kafka_streams_monitoring>. Note that not all
of
>> them
>>>> are
>>>>>>> turned on by default.
>>>>>>> 
>>>>>>> We have several benchmarks that run nightly to monitor streams
>>>>>>> performance. They all stem from the SimpleBenchmark.java
>> benchmark.
>>> In
>>>>>>> addition, their results are published nightly here
>>>>>>> http://testing.confluent.io <http://testing.confluent.io/>,
>> (e.g.,
>>>>> under
>>>>>>> the trunk results). E.g., looking at today's results:
>>>>>>> http://confluent-kafka-system-test-results.s3-us-west-2.
>>>>>>> amazonaws.com/2017-03-02--001.1488449554--apache--trunk--
>>>>>>> ef92bb4/report.html <http://confluent-kafka-
>>>> system-test-results.s3-us-
>>>>>>> west-2.amazonaws.com/2017-03-02--001.1488449554--apache--
>>>>>>> trunk--ef92bb4/report.html>
>>>>>>> (if you search for "benchmarks.streams") you'll see results from
a
>>>>> series
>>>>>>> of benchmarks, ranging from simply consuming, to simple topologies
>>>> with
>>>>> a
>>>>>>> source and sink, to joins and count aggregate. These run on AWS
>>>> nightly,
>>>>>>> but you can also run manually on your setup.
>>>>>>> 
>>>>>>> In addition, programmatically the code can check the
>>>>> KafkaStreams.state()
>>>>>>> and register listeners for when the state changes. For example,
>> the
>>>>> state
>>>>>>> can change from "running" to "rebalancing".
>>>>>>> 
>>>>>>> It is likely we'll need more metrics moving forward and would
be
>>> great
>>>>> to
>>>>>>> get feedback from the community.
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Eno
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On 2 Mar 2017, at 11:54, Sachin Mittal <sjmittal@gmail.com>
>> wrote:
>>>>>>>> 
>>>>>>>> Hello All,
>>>>>>>> I had few questions regarding monitoring of kafka streams
>>> application
>>>>> and
>>>>>>>> what are some important metrics we should collect in our
case.
>>>>>>>> 
>>>>>>>> Just a brief overview, we have a single thread application
>>> (0.10.1.1)
>>>>>>>> reading from single partition topic and it is working all
fine.
>>>>>>>> Then we have same application (using 0.10.2.0) multi threaded
>> with
>>> 4
>>>>>>>> threads per machine and 3 machines cluster setup reading
for same
>>> but
>>>>>>>> partitioned topic (12 partitions).
>>>>>>>> Thus we have each thread processing single partition same
case as
>>>>> earlier
>>>>>>>> one.
>>>>>>>> 
>>>>>>>> The new setup also works fine in steady state, but under
load
>>> somehow
>>>>> it
>>>>>>>> triggers frequent re-balance and then we run into all sort
of
>>> issues
>>>>> like
>>>>>>>> stream thread dying due to CommitFailedException or entering
into
>>>>>>> deadlock
>>>>>>>> state.
>>>>>>>> After a while we restart all the instances then it works
fine
>> for a
>>>>> while
>>>>>>>> and again we get the same problem and it goes on.
>>>>>>>> 
>>>>>>>> 1. So just to monitor, like when first thread fails what
would be
>>>> some
>>>>>>>> important metrics we should be collecting to get some sense
of
>>> whats
>>>>>>> going
>>>>>>>> on?
>>>>>>>> 
>>>>>>>> 2. Is there any metric that tells time elapsed between successive
>>>> poll
>>>>>>>> requests, so we can monitor that?
>>>>>>>> 
>>>>>>>> Also I did monitor rocksdb put and fetch times for these
2
>>> instances
>>>>> and
>>>>>>>> here is the output I get:
>>>>>>>> 0.10.1.1
>>>>>>>> $>get -s  -b kafka.streams:type=stream-
>>>> rocksdb-window-metrics,client-
>>>>>>> id=new-advice-1-StreamThread-1
>>>>>>>> key-table-put-avg-latency-ms
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-advice-1-StreamThread-1:
>>>>>>>> 206431.7497615029
>>>>>>>> $>get -s  -b kafka.streams:type=stream-
>>>> rocksdb-window-metrics,client-
>>>>>>> id=new-advice-1-StreamThread-1
>>>>>>>> key-table-fetch-avg-latency-ms
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-advice-1-StreamThread-1:
>>>>>>>> 2595394.2746129474
>>>>>>>> $>get -s  -b kafka.streams:type=stream-
>>>> rocksdb-window-metrics,client-
>>>>>>> id=new-advice-1-StreamThread-1
>>>>>>>> key-table-put-qps
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-advice-1-StreamThread-1:
>>>>>>>> 232.86299499317252
>>>>>>>> $>get -s  -b kafka.streams:type=stream-
>>>> rocksdb-window-metrics,client-
>>>>>>> id=new-advice-1-StreamThread-1
>>>>>>>> key-table-fetch-qps
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-advice-1-StreamThread-1:
>>>>>>>> 373.61071016166284
>>>>>>>> 
>>>>>>>> Same values for 0.10.2.0 I get
>>>>>>>> $>get -s -b kafka.streams:type=stream-
>>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>> StreamThread-1
>>>>>>>> key-table-put-latency-avg
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>>> StreamThread-1:
>>>>>>>> 1199859.5535022356
>>>>>>>> $>get -s -b kafka.streams:type=stream-
>>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>> StreamThread-1
>>>>>>>> key-table-fetch-latency-avg
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>>> StreamThread-1:
>>>>>>>> 3679340.80748852
>>>>>>>> $>get -s -b kafka.streams:type=stream-
>>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>> StreamThread-1
>>>>>>>> key-table-put-rate
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>>> StreamThread-1:
>>>>>>>> 56.134778706069184
>>>>>>>> $>get -s -b kafka.streams:type=stream-
>>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>> StreamThread-1
>>>>>>>> key-table-fetch-rate
>>>>>>>> #mbean = kafka.streams:type=stream-
>> rocksdb-window-metrics,client-
>>>>>>>> id=new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-
>>>>> StreamThread-1:
>>>>>>>> 136.10721427931827
>>>>>>>> 
>>>>>>>> I notice that result in 10.2.0 is much worse than same for
10.1.1
>>>>>>>> 
>>>>>>>> I would like to know
>>>>>>>> 1. Is there any benchmark on rocksdb as at what rate/latency
it
>>>> should
>>>>> be
>>>>>>>> doing put/fetch operations.
>>>>>>>> 
>>>>>>>> 2. What could be the cause of inferior numbers in 10.2.0,
is it
>>>> because
>>>>>>>> this application is also running three other threads doing
the
>> same
>>>>>>> thing.
>>>>>>>> 
>>>>>>>> 3. Also whats with the name new-part-advice-d1094e71-
>>>>>>>> 0f59-45e8-98f4-477f9444aa91-StreamThread-1
>>>>>>>>  I wanted to put this as a part of my cronjob, so why can't
we
>>> have
>>>>>>>> simpler name like we have in 10.1.1, so it is easy to write
the
>>>> script.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Sachin
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> -- Guozhang
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> -- Guozhang
>> 


Mime
View raw message