ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikita Amelchev <nsamelc...@gmail.com>
Subject Re: Tool for performance statistics reports
Date Thu, 11 Jun 2020 08:20:05 GMT
Hello,

> Can you please share more info on how we can use the profiling tool with
> ignite-extensions modules?

I have updated the module readme file. You can find instructions
there. For now, I am working on review fixes and marked PR as a draft.

чт, 11 июн. 2020 г. в 03:21, Saikat Maitra <saikat.maitra@gmail.com>:
>
> Hello Nikita,
>
> I observed we have an open PR in ignite repo for this feature with
> different set of changes compared to ignite extensions repo.
>
>
> apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>
>
> https://github.com/apache/ignite-extensions/pull/16
>
> Can you please share more info on how we can use the profiling tool with
> ignite-extensions modules?
>
>
> Regards,
>
> Saikat
>
> On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <nizhikov@apache.org> wrote:
>
> > Hello, Alexey.
> >
> > Thanks for the review.
> >
> > My understanding if the following:
> >
> > We will have 3 in-depth tool to find issues in cluster:
> >
> > 1. Metrics + System views - data that describe Ignite entities very
> > high-level.
> >
> > 2. Profiling - tool to know what specific query of transactions are slow.
> > In many cases, this knowledge is enough to fix the issue(rewrite query,
> > redesign transactions flow, etc)
> >
> > 3. Tracing - tool to know why one of 1000 of the same queries was slow.
> > The most detailed view of the Ignite internal processes.
> >
> > > For example, a user would not be able to match a long task with a long
> > job in that task.
> >
> > This is not true.
> > Profiling report will aggregate data from all nodes.
> > So there will be both
> >
> >  * summary time of the task
> >  * time of the each job in the task.
> >
> >
> > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <alexey.goncharuk@gmail.com>
> > написал(а):
> > >
> > > Nikita, Igniters,
> > >
> > > I left a few comments on the tool itself in the PR.
> > >
> > > However, I would like to reiterate and discuss why a user would prefer to
> > > use the profiling tool over tracing? Profiling tool only captures very
> > > high-level details of the operations (a single cache operation, for
> > > example), and does not interconnect operations happened on different
> > nodes.
> > > For example, a user would not be able to match a long task with a long
> > job
> > > in that task. In other words, profiling data is always a subset of data
> > > collected by tracing.
> > >
> > > Maybe it makes sense to adopt local log file approach to write spans so
> > we
> > > can process those spans later to build a report?
> > >
> > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <nsamelchev@gmail.com>:
> > >
> > >> Hi, Igniters.
> > >>
> > >> I have implemented cluster profiling and tool to build the performance
> > >> report. It's ready to be reviewed. [1, 2]
> > >>
> > >> Profiling can be managed by JMX bean. I have plans to implement it to
> > >> control.sh also.
> > >>
> > >> Nodes write statistics to the temporary off heap buffer and then one
> > >> thread flushes to the profiling files. The write mechanics and format
> > >> is like WAL logging.
> > >> The report contains the following statistics:
> > >> - nodes and caches info
> > >> - cache operations and transaction statistics
> > >> - SQL and scan queries statistics (include logical and physical reads
> > per
> > >> query)
> > >> - tasks and jobs statistics.
> > >>
> > >> More details in the IEP [3].
> > >>
> > >> [1] https://github.com/apache/ignite/pull/7693
> > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> > >> [3]
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>
> > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
> > slava.koptilin@gmail.com>:
> > >>>
> > >>> Hello Nikolay,
> > >>>
> > >>>> Who deprecated visor and when? Maybe I miss something?
> > >>> On the one hand, there was technically no community consensus that
this
> > >>> tool should be obsolete.
> > >>> On the other hand, my opinion based on the following topic:
> > >>>
> > >>
> > http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> > >>> Moreover, it seems to me, currently, the control utility is widely
used
> > >> and
> > >>> actively developed, instead of the visor.
> > >>>
> > >>>> It's true that, for now, Ignite doesn't have "tool strategy" I
think
> > >> it's
> > >>> a big issue from the user's point of view.
> > >>> I absolutely agree with that.
> > >>>
> > >>>> We should solve it in the nearest time. Feel free to start this
> > >> activity
> > >>> I have no plan at the moment. However, at the first stage, we could
> > >>> understand the difference between visor and control utility.
> > >>> All useful features from visor should be moved/implemented in control
> > >>> utility and after that visor tool and should be marked as
> > >>> deprecated/obsoleted.
> > >>>
> > >>>> You need to throw in control.sh also, which does some kind of
> > >> statistics
> > >>> too, such as idle_verify.
> > >>>> Please, clarify your idea:
> > >>>>   We should use some info from control.sh to the report?
> > >>>>   The report should be generated by some control.sh subcommand?
> > >>> If I am not mistaken, the oracle database has AWR tool (mentioned on
> > the
> > >>> IEP page) which is a command-line utility that generates HTML reports.
> > >>> I like this idea and I think this is a good approach that can be
> > realized
> > >>> in the control utility.
> > >>> If we have a case that cannot be implemented in this way, we have to
> > >>> clearly states the difference between these tools so as not to confuse
> > >> our
> > >>> users.
> > >>> What do you think?
> > >>>
> > >>> Thanks,
> > >>> Slava.
> > >>>
> > >>>
> > >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <nizhikov@apache.org>:
> > >>>
> > >>>> Hello, Slava, Ilya, Denis.
> > >>>>
> > >>>> Thanks for joining this discussion!
> > >>>>
> > >>>>> - visor (which is deprecated)
> > >>>>
> > >>>> Who deprecated visor and when?
> > >>>> Maybe I miss something?
> > >>>>
> > >>>>> - web-console (to be honest, I don't quite understand the status
of
> > >> this
> > >>>> tool)
> > >>>>
> > >>>> +1.
> > >>>>
> > >>>>> I am not against the new tool, I just want to understand the
> > >> motivation
> > >>>> to not improve the existing sub-projects.
> > >>>>
> > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
> > >>>> I think it's a big issue from the user's point of view.
> > >>>> We should solve it in the nearest time.
> > >>>> Feel free to start this activity.
> > >>>>
> > >>>>> - new ignite-profiling (which is a monitoring tool as well,
judging
> > >> by
> > >>>> the provided link [1] )
> > >>>>
> > >>>> The general idea is the following:
> > >>>>
> > >>>> 1. We should have some profiling mechanism that will generate a
> > >> node-local
> > >>>> event log
> > >>>> 2. We should have a tool that can export events to some third-party
> > >>>> system. This system can be an Elastic Search(Kibana) or Ignite
> > >> performance
> > >>>> report or Kafka log, whatever.
> > >>>> 3. Ignite performance report, in the first release, should be a
> > >> "static"
> > >>>> tool.
> > >>>>    This means we take static logs(that is not rewritten in the
> > >> analysis
> > >>>> time) and feed them in the script(or tool or control.sh, whatever)
> > >>>>    The script produces static report that can be used for overall
> > >>>> performance analysis.
> > >>>>
> > >>>> The primary users of this report is a developer of Ignite based
> > >>>> applications and performance engineers.
> > >>>>
> > >>>> Ilya,
> > >>>>
> > >>>>> You need to throw in control.sh also, which does some kind
of
> > >> statistics
> > >>>> too, such as idle_verify.
> > >>>>
> > >>>> Please, clarify your idea:
> > >>>>    We should use some info from control.sh to the report?
> > >>>>    The report should be generated by some control.sh subcommand?
> > >>>>
> > >>>>
> > >>>> Denis,
> > >>>>
> > >>>>> Speaking of the probes/statistics collection approach, is it
> > >> supposed to
> > >>>> reuse tracing capabilities that are to be added as part of IEP-35?
> > >>>>
> > >>>> For now, we don't have any results of tracing development available
in
> > >>>> Apache Ignite.
> > >>>> Hopefully, we got some in a couple of weeks.
> > >>>> After it, we can start a discussion of how to merge two improvements.
> > >>>>
> > >>>>
> > >>>>
> > >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <dmagda@apache.org>
> > >> написал(а):
> > >>>>>
> > >>>>>>
> > >>>>>> Tracing is more deeply takes statistics. If it will be
possible,
> > >> I'm for
> > >>>>>> reuse.
> > >>>>>
> > >>>>>
> > >>>>> Looks like we need to sync up on these activities/initiatives
to
> > >> ensure
> > >>>> we
> > >>>>> don't do a duplicate job. If you think a separate discussion
is
> > >> necessary
> > >>>>> let's kick it off.
> > >>>>>
> > >>>>> -
> > >>>>> Denis
> > >>>>>
> > >>>>>
> > >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> > >> nsamelchev@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Denis, Ilya,
> > >>>>>>
> > >>>>>> I will try to integrate profiling functionality into control.sh
> > >> utility.
> > >>>>>>
> > >>>>>>> Speaking of the probes/statistics collection approach,
is it
> > >> supposed
> > >>>> to
> > >>>>>>> reuse tracing capabilities that are to be added as
part of IEP-35?
> > >>>>>> Tracing is more deeply takes statistics. If it will be
possible,
> > >> I'm for
> > >>>>>> reuse.
> > >>>>>>
> > >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
> > >> ilya.kasnacheev@gmail.com
> > >>>>> :
> > >>>>>>>
> > >>>>>>> Hello!
> > >>>>>>>
> > >>>>>>> I suggest that it's one of the places where it could
be put
> > >> instead of
> > >>>>>>> adding a new tool.
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> --
> > >>>>>>> Ilya Kasnacheev
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev
<nsamelchev@gmail.com
> > >>> :
> > >>>>>>>
> > >>>>>>>> Ilya,
> > >>>>>>>>
> > >>>>>>>> You suggest using control.sh to build the report?
> > >>>>>>>>
> > >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev
<
> > >>>>>> ilya.kasnacheev@gmail.com>:
> > >>>>>>>>>
> > >>>>>>>>> Hello!
> > >>>>>>>>>
> > >>>>>>>>> You need to throw in control.sh also, which
does some kind of
> > >>>>>> statistics
> > >>>>>>>>> too, such as idle_verify.
> > >>>>>>>>>
> > >>>>>>>>> Regards,
> > >>>>>>>>> --
> > >>>>>>>>> Ilya Kasnacheev
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав
Коптилин <
> > >>>>>> slava.koptilin@gmail.com
> > >>>>>>>>> :
> > >>>>>>>>>
> > >>>>>>>>>> Hello Nikita,
> > >>>>>>>>>>
> > >>>>>>>>>> Perhaps, I am missing something...
> > >>>>>>>>>> Apache Ignite already has a web-console
tool. Do we want to
> > >>>>>> improve the
> > >>>>>>>>>> existing tool instead of creating a new
one?
> > >>>>>>>>>> It seems to me, this can be confusing for
users.
> > >>>>>>>>>> - visor (which is deprecated)
> > >>>>>>>>>> - web-console (to be honest, I don't quite
understand the status
> > >>>>>> of
> > >>>>>>>> this
> > >>>>>>>>>> tool)
> > >>>>>>>>>> - new ignite-profiling (which is a monitoring
tool as well,
> > >>>>>> judging
> > >>>>>>>> by the
> > >>>>>>>>>> provided link [1] )
> > >>>>>>>>>>
> > >>>>>>>>>> I am not against the new tool, I just want
to understand the
> > >>>>>>>> motivation to
> > >>>>>>>>>> not improve the existing sub-projects.
> > >>>>>>>>>>
> > >>>>>>>>>> [1]
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> S.
> > >>>>>>>>>>
> > >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita
Amelchev <
> > >> nsamelchev@gmail.com
> > >>>>>>> :
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi, Igniters.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm working on cluster profiling and
the tool for creating a
> > >>>>>>>>>>> performance report. [1] I have prepared
PoC based on
> > >> performance
> > >>>>>>>>>>> logging to a separate category of Ignite
log. The report
> > >>>>>> contains:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Cache operations and its distribution
by types [2]
> > >>>>>>>>>>> - Transactions and histogram of durations
[3]
> > >>>>>>>>>>> - SQL and Scan query statistics, top
of slowest queries,
> > >> logical
> > >>>>>> and
> > >>>>>>>>>>> physical reads by query [4]
> > >>>>>>>>>>> - Compute statistics, top of slowest
tasks and their jobs [5]
> > >>>>>>>>>>> Soon I will add:
> > >>>>>>>>>>> - Topology and Ignite versions info
> > >>>>>>>>>>> - Client ID in case of operations from
clients
> > >>>>>>>>>>>
> > >>>>>>>>>>> For now, I'm developing a binary logging
format to reduce the
> > >>>>>> effect
> > >>>>>>>>>>> on performance. I'll try to reuse Ignite
mechanisms.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I would like to hear suggestions for
the profiling and the
> > >>>>>>>> performance
> > >>>>>>>>>>> report.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [1]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> > >>>>>>>>>>> [2]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> > >>>>>>>>>>> [3]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> > >>>>>>>>>>> [4]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > >>>>>>>>>>> [5]
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Best wishes,
> > >>>>>>>>>>> Amelchev Nikita
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Best wishes,
> > >>>>>>>> Amelchev Nikita
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best wishes,
> > >>>>>> Amelchev Nikita
> > >>>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best wishes,
> > >> Amelchev Nikita
> > >>
> >
> >



-- 
Best wishes,
Amelchev Nikita

Mime
View raw message