ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ткаленко кирилл <tkalkir...@yandex.ru>
Subject Re: [DISCUSSION] Add index rebuild time metrics
Date Mon, 10 Aug 2020 17:53:26 GMT
Hi, Ivan!

For this you can use org.apache.ignite.cache.CacheMetrics#IsIndexRebuildInProgress
> How can a local number of processed keys can help us to understand when
> index rebuild will be finished?

This metric can be used only for local node, to get size of cache use org.apache.ignite.internal.processors.cache.CacheMetricsImpl#getCacheSize.
> We can't compare metric value with cache.size(). First one is node-local,
> while cache size covers all partitions in the cluster.

If there is a lot of data in node that can be rebuilt, percentage may change very rarely and
may not give an estimate of how much time is left. If we see for example that 50_000 keys
are rebuilt once a minute, and we have 1_000_000_000 keys, then we can have an approximate
estimate. What do you think of that?
> I find one single metric much more usable. It would be perfect if metric
> value is represented in percentage, e.g. current progress of local node
> index rebuild is 60%.

10.08.2020, 19:11, "Ivan Rakov" <ivan.glukos@gmail.com>:
> Folks,
>
> Sorry for coming late to the party. I've taken a look at this issue during
> review.
>
> How can a local number of processed keys can help us to understand when
> index rebuild will be finished?
> We can't compare metric value with cache.size(). First one is node-local,
> while cache size covers all partitions in the cluster.
> Also, I don't understand why we need to keep separate metrics for all
> caches. Of course, the metric becomes more fair, but obviously harder to
> make conclusions on whether "the index rebuild" process is over (and the
> cluster is ready to process queries quickly).
>
> I find one single metric much more usable. It would be perfect if metric
> value is represented in percentage, e.g. current progress of local node
> index rebuild is 60%.
>
> --
> Best regards,
> Ivan
>
> On Fri, Jul 24, 2020 at 1:35 PM Stanislav Lukyanov <stanlukyanov@gmail.com>
> wrote:
>
>>  Got it. I thought that index building and index rebuilding are essentially
>>  the same,
>>  but now I see that they are different: index rebuilding cares about all
>>  indexes at once while index building cares about particular ones.
>>
>>  Kirill's approach sounds good.
>>
>>  Stan
>>
>>  > On 20 Jul 2020, at 14:54, Alexey Goncharuk <alexey.goncharuk@gmail.com>
>>  wrote:
>>  >
>>  > Stan,
>>  >
>>  > Currently we never build indexes one-by-one - we always use a cache data
>>  > row visitor which either updates all indexes (see
>>  IndexRebuildFullClosure)
>>  > or updates a set of all indexes that need to catch up (see
>>  > IndexRebuildPartialClosure). GIven that, I do not see any need for
>>  > per-index rebuild status as this status will be updated for all outdated
>>  > indexes simultaneously.
>>  >
>>  > Kirill's approach for the total number of processed keys per cache seems
>>  > reasonable to me.
>>  >
>>  > --AG
>>  >
>>  > пт, 3 июл. 2020 г. в 10:12, ткаленко кирилл <tkalkirill@yandex.ru>:
>>  >
>>  >> Hi, Stan!
>>  >>
>>  >> Perhaps it is worth clarifying what exactly I wanted to say.
>>  >> Now we have 2 processes: building and rebuilding indexes.
>>  >>
>>  >> At moment, we have some metrics for rebuilding indexes:
>>  >> "IsIndexRebuildInProgress", "IndexBuildCountPartitionsLeft".
>>  >>
>>  >> I suggest adding another metric "Indexrebuildkeyprocessed", which will
>>  >> allow you to determine how many records are left to rebuild for cache.
>>  >>
>>  >> I think your comments are more about building an index that may need
>>  more
>>  >> metrics, but I think you should do it in a separate ticket.
>>  >>
>>  >> 03.07.2020, 03:09, "Stanislav Lukyanov" <stanlukyanov@gmail.com>:
>>  >>> If multiple indexes are to be built "number of indexed keys" metric
may
>>  >> be misleading.
>>  >>>
>>  >>> As a cluster admin, I'd like to know:
>>  >>> - Are all indexes ready on a node?
>>  >>> - How many indexes are to be built?
>>  >>> - How much resources are used by the index building (how many threads
>>  >> are used)?
>>  >>> - Which index(es?) is being built right now?
>>  >>> - How much time until the current (single) index building finishes?
>>  Here
>>  >> "time" can be a lot of things: partitions, entries, percent of the
>>  cache,
>>  >> minutes and hours
>>  >>> - How much time until all indexes are built?
>>  >>> - How much does it take to build each of my indexes / a single index
of
>>  >> my cache on average?
>>  >>>
>>  >>> I think we need a set of metrics and/or log messages to solve all
of
>>  >> these questions.
>>  >>> I imaging something like:
>>  >>> - numberOfIndexesToBuild
>>  >>> - a standard set of metrics on the index building thread pool (do
we
>>  >> already have it?)
>>  >>> - currentlyBuiltIndexName (assuming we only build one at a time which
>>  is
>>  >> probably not true)
>>  >>> - for the "time" metrics I think percentage might be the best as it's
>>  >> the easiest to understand; we may add multiple metrics though.
>>  >>> - For "time per each index" I'd add detailed log messages stating
how
>>  >> long did it take to build a particular index
>>  >>>
>>  >>> Thanks,
>>  >>> Stan
>>  >>>
>>  >>>> On 26 Jun 2020, at 12:49, ткаленко кирилл <tkalkirill@yandex.ru>
>>  >> wrote:
>>  >>>>
>>  >>>> Hi, Igniters.
>>  >>>>
>>  >>>> I would like to know if it is possible to estimate how much the
index
>>  >> rebuild will take?
>>  >>>>
>>  >>>> At the moment, I have found the following metrics [1] and [2]
and
>>  >> since the rebuild is based on caches, I think it would be useful to know
>>  >> how many records are processed in indexing. This way we can estimate how
>>  >> long we have to wait for the index to be rebuilt by subtracting [3] and
>>  how
>>  >> many records are indexed.
>>  >>>>
>>  >>>> I think we should add this metric [4].
>>  >>>>
>>  >>>> Comments, suggestions?
>>  >>>>
>>  >>>> [1] - https://issues.apache.org/jira/browse/IGNITE-12184
>>  >>>> [2] -
>>  >>
>>  org.apache.ignite.internal.processors.cache.CacheGroupMetricsImpl#idxBuildCntPartitionsLeft
>>  >>>> [3] - org.apache.ignite.cache.CacheMetrics#getCacheSize
>>  >>>> [4] - org.apache.ignite.cache.CacheMetrics#getNumberIndexedKeys
>>  >>

Mime
View raw message