ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: Tool for performance statistics reports
Date Sat, 13 Jun 2020 01:45:31 GMT
Hello Nikita,

Thank you for sharing the information, it is very helpful.

Regards,
Saikat

On Thu, Jun 11, 2020 at 3:20 AM Nikita Amelchev <nsamelchev@gmail.com>
wrote:

> Hello,
>
> > Can you please share more info on how we can use the profiling tool with
> > ignite-extensions modules?
>
> I have updated the module readme file. You can find instructions
> there. For now, I am working on review fixes and marked PR as a draft.
>
> чт, 11 июн. 2020 г. в 03:21, Saikat Maitra <saikat.maitra@gmail.com>:
> >
> > Hello Nikita,
> >
> > I observed we have an open PR in ignite repo for this feature with
> > different set of changes compared to ignite extensions repo.
> >
> >
> > apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>
> >
> > https://github.com/apache/ignite-extensions/pull/16
> >
> > Can you please share more info on how we can use the profiling tool with
> > ignite-extensions modules?
> >
> >
> > Regards,
> >
> > Saikat
> >
> > On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <nizhikov@apache.org>
> wrote:
> >
> > > Hello, Alexey.
> > >
> > > Thanks for the review.
> > >
> > > My understanding if the following:
> > >
> > > We will have 3 in-depth tool to find issues in cluster:
> > >
> > > 1. Metrics + System views - data that describe Ignite entities very
> > > high-level.
> > >
> > > 2. Profiling - tool to know what specific query of transactions are
> slow.
> > > In many cases, this knowledge is enough to fix the issue(rewrite query,
> > > redesign transactions flow, etc)
> > >
> > > 3. Tracing - tool to know why one of 1000 of the same queries was slow.
> > > The most detailed view of the Ignite internal processes.
> > >
> > > > For example, a user would not be able to match a long task with a
> long
> > > job in that task.
> > >
> > > This is not true.
> > > Profiling report will aggregate data from all nodes.
> > > So there will be both
> > >
> > >  * summary time of the task
> > >  * time of the each job in the task.
> > >
> > >
> > > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <
> alexey.goncharuk@gmail.com>
> > > написал(а):
> > > >
> > > > Nikita, Igniters,
> > > >
> > > > I left a few comments on the tool itself in the PR.
> > > >
> > > > However, I would like to reiterate and discuss why a user would
> prefer to
> > > > use the profiling tool over tracing? Profiling tool only captures
> very
> > > > high-level details of the operations (a single cache operation, for
> > > > example), and does not interconnect operations happened on different
> > > nodes.
> > > > For example, a user would not be able to match a long task with a
> long
> > > job
> > > > in that task. In other words, profiling data is always a subset of
> data
> > > > collected by tracing.
> > > >
> > > > Maybe it makes sense to adopt local log file approach to write spans
> so
> > > we
> > > > can process those spans later to build a report?
> > > >
> > > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <nsamelchev@gmail.com>:
> > > >
> > > >> Hi, Igniters.
> > > >>
> > > >> I have implemented cluster profiling and tool to build the
> performance
> > > >> report. It's ready to be reviewed. [1, 2]
> > > >>
> > > >> Profiling can be managed by JMX bean. I have plans to implement it
> to
> > > >> control.sh also.
> > > >>
> > > >> Nodes write statistics to the temporary off heap buffer and then one
> > > >> thread flushes to the profiling files. The write mechanics and
> format
> > > >> is like WAL logging.
> > > >> The report contains the following statistics:
> > > >> - nodes and caches info
> > > >> - cache operations and transaction statistics
> > > >> - SQL and scan queries statistics (include logical and physical
> reads
> > > per
> > > >> query)
> > > >> - tasks and jobs statistics.
> > > >>
> > > >> More details in the IEP [3].
> > > >>
> > > >> [1] https://github.com/apache/ignite/pull/7693
> > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> > > >> [3]
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>
> > > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин
<
> > > slava.koptilin@gmail.com>:
> > > >>>
> > > >>> Hello Nikolay,
> > > >>>
> > > >>>> Who deprecated visor and when? Maybe I miss something?
> > > >>> On the one hand, there was technically no community consensus
that
> this
> > > >>> tool should be obsolete.
> > > >>> On the other hand, my opinion based on the following topic:
> > > >>>
> > > >>
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> > > >>> Moreover, it seems to me, currently, the control utility is widely
> used
> > > >> and
> > > >>> actively developed, instead of the visor.
> > > >>>
> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
I
> think
> > > >> it's
> > > >>> a big issue from the user's point of view.
> > > >>> I absolutely agree with that.
> > > >>>
> > > >>>> We should solve it in the nearest time. Feel free to start
this
> > > >> activity
> > > >>> I have no plan at the moment. However, at the first stage, we
could
> > > >>> understand the difference between visor and control utility.
> > > >>> All useful features from visor should be moved/implemented in
> control
> > > >>> utility and after that visor tool and should be marked as
> > > >>> deprecated/obsoleted.
> > > >>>
> > > >>>> You need to throw in control.sh also, which does some kind
of
> > > >> statistics
> > > >>> too, such as idle_verify.
> > > >>>> Please, clarify your idea:
> > > >>>>   We should use some info from control.sh to the report?
> > > >>>>   The report should be generated by some control.sh subcommand?
> > > >>> If I am not mistaken, the oracle database has AWR tool (mentioned
> on
> > > the
> > > >>> IEP page) which is a command-line utility that generates HTML
> reports.
> > > >>> I like this idea and I think this is a good approach that can
be
> > > realized
> > > >>> in the control utility.
> > > >>> If we have a case that cannot be implemented in this way, we have
> to
> > > >>> clearly states the difference between these tools so as not to
> confuse
> > > >> our
> > > >>> users.
> > > >>> What do you think?
> > > >>>
> > > >>> Thanks,
> > > >>> Slava.
> > > >>>
> > > >>>
> > > >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <nizhikov@apache.org
> >:
> > > >>>
> > > >>>> Hello, Slava, Ilya, Denis.
> > > >>>>
> > > >>>> Thanks for joining this discussion!
> > > >>>>
> > > >>>>> - visor (which is deprecated)
> > > >>>>
> > > >>>> Who deprecated visor and when?
> > > >>>> Maybe I miss something?
> > > >>>>
> > > >>>>> - web-console (to be honest, I don't quite understand
the status
> of
> > > >> this
> > > >>>> tool)
> > > >>>>
> > > >>>> +1.
> > > >>>>
> > > >>>>> I am not against the new tool, I just want to understand
the
> > > >> motivation
> > > >>>> to not improve the existing sub-projects.
> > > >>>>
> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
> > > >>>> I think it's a big issue from the user's point of view.
> > > >>>> We should solve it in the nearest time.
> > > >>>> Feel free to start this activity.
> > > >>>>
> > > >>>>> - new ignite-profiling (which is a monitoring tool as
well,
> judging
> > > >> by
> > > >>>> the provided link [1] )
> > > >>>>
> > > >>>> The general idea is the following:
> > > >>>>
> > > >>>> 1. We should have some profiling mechanism that will generate
a
> > > >> node-local
> > > >>>> event log
> > > >>>> 2. We should have a tool that can export events to some
> third-party
> > > >>>> system. This system can be an Elastic Search(Kibana) or Ignite
> > > >> performance
> > > >>>> report or Kafka log, whatever.
> > > >>>> 3. Ignite performance report, in the first release, should
be a
> > > >> "static"
> > > >>>> tool.
> > > >>>>    This means we take static logs(that is not rewritten in
the
> > > >> analysis
> > > >>>> time) and feed them in the script(or tool or control.sh, whatever)
> > > >>>>    The script produces static report that can be used for
overall
> > > >>>> performance analysis.
> > > >>>>
> > > >>>> The primary users of this report is a developer of Ignite
based
> > > >>>> applications and performance engineers.
> > > >>>>
> > > >>>> Ilya,
> > > >>>>
> > > >>>>> You need to throw in control.sh also, which does some
kind of
> > > >> statistics
> > > >>>> too, such as idle_verify.
> > > >>>>
> > > >>>> Please, clarify your idea:
> > > >>>>    We should use some info from control.sh to the report?
> > > >>>>    The report should be generated by some control.sh subcommand?
> > > >>>>
> > > >>>>
> > > >>>> Denis,
> > > >>>>
> > > >>>>> Speaking of the probes/statistics collection approach,
is it
> > > >> supposed to
> > > >>>> reuse tracing capabilities that are to be added as part of
IEP-35?
> > > >>>>
> > > >>>> For now, we don't have any results of tracing development
> available in
> > > >>>> Apache Ignite.
> > > >>>> Hopefully, we got some in a couple of weeks.
> > > >>>> After it, we can start a discussion of how to merge two
> improvements.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <dmagda@apache.org>
> > > >> написал(а):
> > > >>>>>
> > > >>>>>>
> > > >>>>>> Tracing is more deeply takes statistics. If it will
be possible,
> > > >> I'm for
> > > >>>>>> reuse.
> > > >>>>>
> > > >>>>>
> > > >>>>> Looks like we need to sync up on these activities/initiatives
to
> > > >> ensure
> > > >>>> we
> > > >>>>> don't do a duplicate job. If you think a separate discussion
is
> > > >> necessary
> > > >>>>> let's kick it off.
> > > >>>>>
> > > >>>>> -
> > > >>>>> Denis
> > > >>>>>
> > > >>>>>
> > > >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> > > >> nsamelchev@gmail.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Denis, Ilya,
> > > >>>>>>
> > > >>>>>> I will try to integrate profiling functionality into
control.sh
> > > >> utility.
> > > >>>>>>
> > > >>>>>>> Speaking of the probes/statistics collection approach,
is it
> > > >> supposed
> > > >>>> to
> > > >>>>>>> reuse tracing capabilities that are to be added
as part of
> IEP-35?
> > > >>>>>> Tracing is more deeply takes statistics. If it will
be possible,
> > > >> I'm for
> > > >>>>>> reuse.
> > > >>>>>>
> > > >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev
<
> > > >> ilya.kasnacheev@gmail.com
> > > >>>>> :
> > > >>>>>>>
> > > >>>>>>> Hello!
> > > >>>>>>>
> > > >>>>>>> I suggest that it's one of the places where it
could be put
> > > >> instead of
> > > >>>>>>> adding a new tool.
> > > >>>>>>>
> > > >>>>>>> Regards,
> > > >>>>>>> --
> > > >>>>>>> Ilya Kasnacheev
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev
<
> nsamelchev@gmail.com
> > > >>> :
> > > >>>>>>>
> > > >>>>>>>> Ilya,
> > > >>>>>>>>
> > > >>>>>>>> You suggest using control.sh to build the
report?
> > > >>>>>>>>
> > > >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev
<
> > > >>>>>> ilya.kasnacheev@gmail.com>:
> > > >>>>>>>>>
> > > >>>>>>>>> Hello!
> > > >>>>>>>>>
> > > >>>>>>>>> You need to throw in control.sh also,
which does some kind of
> > > >>>>>> statistics
> > > >>>>>>>>> too, such as idle_verify.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> --
> > > >>>>>>>>> Ilya Kasnacheev
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав
Коптилин <
> > > >>>>>> slava.koptilin@gmail.com
> > > >>>>>>>>> :
> > > >>>>>>>>>
> > > >>>>>>>>>> Hello Nikita,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Perhaps, I am missing something...
> > > >>>>>>>>>> Apache Ignite already has a web-console
tool. Do we want to
> > > >>>>>> improve the
> > > >>>>>>>>>> existing tool instead of creating
a new one?
> > > >>>>>>>>>> It seems to me, this can be confusing
for users.
> > > >>>>>>>>>> - visor (which is deprecated)
> > > >>>>>>>>>> - web-console (to be honest, I don't
quite understand the
> status
> > > >>>>>> of
> > > >>>>>>>> this
> > > >>>>>>>>>> tool)
> > > >>>>>>>>>> - new ignite-profiling (which is a
monitoring tool as well,
> > > >>>>>> judging
> > > >>>>>>>> by the
> > > >>>>>>>>>> provided link [1] )
> > > >>>>>>>>>>
> > > >>>>>>>>>> I am not against the new tool, I just
want to understand the
> > > >>>>>>>> motivation to
> > > >>>>>>>>>> not improve the existing sub-projects.
> > > >>>>>>>>>>
> > > >>>>>>>>>> [1]
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> S.
> > > >>>>>>>>>>
> > > >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58,
Nikita Amelchev <
> > > >> nsamelchev@gmail.com
> > > >>>>>>> :
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi, Igniters.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm working on cluster profiling
and the tool for creating
> a
> > > >>>>>>>>>>> performance report. [1] I have
prepared PoC based on
> > > >> performance
> > > >>>>>>>>>>> logging to a separate category
of Ignite log. The report
> > > >>>>>> contains:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Cache operations and its distribution
by types [2]
> > > >>>>>>>>>>> - Transactions and histogram of
durations [3]
> > > >>>>>>>>>>> - SQL and Scan query statistics,
top of slowest queries,
> > > >> logical
> > > >>>>>> and
> > > >>>>>>>>>>> physical reads by query [4]
> > > >>>>>>>>>>> - Compute statistics, top of slowest
tasks and their jobs
> [5]
> > > >>>>>>>>>>> Soon I will add:
> > > >>>>>>>>>>> - Topology and Ignite versions
info
> > > >>>>>>>>>>> - Client ID in case of operations
from clients
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> For now, I'm developing a binary
logging format to reduce
> the
> > > >>>>>> effect
> > > >>>>>>>>>>> on performance. I'll try to reuse
Ignite mechanisms.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I would like to hear suggestions
for the profiling and the
> > > >>>>>>>> performance
> > > >>>>>>>>>>> report.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [1]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > > >>>>>>>>>>> [2]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > > >>>>>>>>>>> [3]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > > >>>>>>>>>>> [4]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > > >>>>>>>>>>> [5]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> --
> > > >>>>>>>>>>> Best wishes,
> > > >>>>>>>>>>> Amelchev Nikita
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> --
> > > >>>>>>>> Best wishes,
> > > >>>>>>>> Amelchev Nikita
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Best wishes,
> > > >>>>>> Amelchev Nikita
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best wishes,
> > > >> Amelchev Nikita
> > > >>
> > >
> > >
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message