flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Richter <s.rich...@data-artisans.com>
Subject Re: Flink State monitoring
Date Fri, 20 Apr 2018 09:53:24 GMT
Hi,

for incremental checkpoints, it is only showing the size of the deltas. It would probably
also be possible to report the full size, but the current reporting and UI is only supporting
to deliver a single value. In general, some things are rather hard to report. For example,
for the heap based backend, is the state size the size of the serialized data or the size
of the heap objects? 
Another remark about key count: the key count is easy to determine for the heap based backend,
but there is no (efficient) method in RocksDb that gives the key count (because of the way
RocksDB works). In this case, afaik, we have the (inefficient) option to iterate all keys
and count or use the (efficient) estimated key count is supported by RocksDB.

Best,
Stefan

> Am 04.01.2018 um 19:23 schrieb Steven Wu <stevenz3wu@gmail.com>:
> 
> Aljoscha/Stefan, 
> 
> if incremental checkpoint is enabled, I assume the "checkpoint size" is only the delta/incremental
size (not the full state size), right?
> 
> Thanks,
> Steven
> 
> 
> On Thu, Jan 4, 2018 at 5:18 AM, Aljoscha Krettek <aljoscha@apache.org <mailto:aljoscha@apache.org>>
wrote:
> Hi,
> 
> I'm afraid there is currently no metrics around state. I see that it's very good to have
so I'm putting it on my list of stuff that we should have at some point.
> 
> One thing that comes to mind is checking the size of checkpoints, which gives you an
indirect way of figuring out how big state is but that's not very exact, i.e. doesn't give
you "number of keys" or some such.
> 
> Best,
> Aljoscha
> 
> > On 20. Dec 2017, at 08:09, Netzer, Liron <liron.netzer@citi.com <mailto:liron.netzer@citi.com>>
wrote:
> >
> > Ufuk, Thanks for replying !
> >
> > Aljoscha, can you please assist with the questions below?
> >
> > Thanks,
> > Liron
> >
> > -----Original Message-----
> > From: Ufuk Celebi [mailto:uce@apache.org <mailto:uce@apache.org>]
> > Sent: Friday, December 15, 2017 3:06 PM
> > To: Netzer, Liron [ICG-IT]
> > Cc: user@flink.apache.org <mailto:user@flink.apache.org>
> > Subject: Re: Flink State monitoring
> >
> > Hey Liron,
> >
> > unfortunately, there are no built-in metrics related to state. In general, exposing
the actual values as metrics is problematic, but exposing summary statistics would be a good
idea. I'm not aware of a good work around at the moment that would work in the general case
(taking into account state restore, etc.).
> >
> > Let me pull in Aljoscha (cc'd) who knows the state backend internals well.
> >
> > @Aljoscha:
> > 1) Are there any plans to expose keyed state related metrics (like number of keys)?
> > 2) Is there a way to work around the lack of these metrics in 1.3?
> >
> > – Ufuk
> >
> > On Thu, Dec 14, 2017 at 10:55 AM, Netzer, Liron <liron.netzer@citi.com <mailto:liron.netzer@citi.com>>
wrote:
> >> Hi group,
> >>
> >>
> >>
> >> We are using Flink keyed state in several operators.
> >>
> >> Is there an easy was to expose the data that is stored in the state, i.e.
> >> the key and the values?
> >>
> >> This is needed for both monitoring as well as debugging. We would like
> >> to understand how many key+values are stored in each state and also to
> >> view the data itself.
> >>
> >> I know that there is the "Queryable state" option, but this is still
> >> in Beta, and doesn't really give us what we want easily.
> >>
> >>
> >>
> >>
> >>
> >> *We are using Flink 1.3.2 with Java.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Liron
> 
> 


Mime
View raw message