ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: Tool for performance statistics reports
Date Thu, 11 Jun 2020 00:21:11 GMT
Hello Nikita,

I observed we have an open PR in ignite repo for this feature with
different set of changes compared to ignite extensions repo.


apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>

https://github.com/apache/ignite-extensions/pull/16

Can you please share more info on how we can use the profiling tool with
ignite-extensions modules?


Regards,

Saikat

On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <nizhikov@apache.org> wrote:

> Hello, Alexey.
>
> Thanks for the review.
>
> My understanding if the following:
>
> We will have 3 in-depth tool to find issues in cluster:
>
> 1. Metrics + System views - data that describe Ignite entities very
> high-level.
>
> 2. Profiling - tool to know what specific query of transactions are slow.
> In many cases, this knowledge is enough to fix the issue(rewrite query,
> redesign transactions flow, etc)
>
> 3. Tracing - tool to know why one of 1000 of the same queries was slow.
> The most detailed view of the Ignite internal processes.
>
> > For example, a user would not be able to match a long task with a long
> job in that task.
>
> This is not true.
> Profiling report will aggregate data from all nodes.
> So there will be both
>
>  * summary time of the task
>  * time of the each job in the task.
>
>
> > 8 июня 2020 г., в 12:52, Alexey Goncharuk <alexey.goncharuk@gmail.com>
> написал(а):
> >
> > Nikita, Igniters,
> >
> > I left a few comments on the tool itself in the PR.
> >
> > However, I would like to reiterate and discuss why a user would prefer to
> > use the profiling tool over tracing? Profiling tool only captures very
> > high-level details of the operations (a single cache operation, for
> > example), and does not interconnect operations happened on different
> nodes.
> > For example, a user would not be able to match a long task with a long
> job
> > in that task. In other words, profiling data is always a subset of data
> > collected by tracing.
> >
> > Maybe it makes sense to adopt local log file approach to write spans so
> we
> > can process those spans later to build a report?
> >
> > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <nsamelchev@gmail.com>:
> >
> >> Hi, Igniters.
> >>
> >> I have implemented cluster profiling and tool to build the performance
> >> report. It's ready to be reviewed. [1, 2]
> >>
> >> Profiling can be managed by JMX bean. I have plans to implement it to
> >> control.sh also.
> >>
> >> Nodes write statistics to the temporary off heap buffer and then one
> >> thread flushes to the profiling files. The write mechanics and format
> >> is like WAL logging.
> >> The report contains the following statistics:
> >> - nodes and caches info
> >> - cache operations and transaction statistics
> >> - SQL and scan queries statistics (include logical and physical reads
> per
> >> query)
> >> - tasks and jobs statistics.
> >>
> >> More details in the IEP [3].
> >>
> >> [1] https://github.com/apache/ignite/pull/7693
> >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> >> [3]
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>
> >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
> slava.koptilin@gmail.com>:
> >>>
> >>> Hello Nikolay,
> >>>
> >>>> Who deprecated visor and when? Maybe I miss something?
> >>> On the one hand, there was technically no community consensus that this
> >>> tool should be obsolete.
> >>> On the other hand, my opinion based on the following topic:
> >>>
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
> >>> Moreover, it seems to me, currently, the control utility is widely used
> >> and
> >>> actively developed, instead of the visor.
> >>>
> >>>> It's true that, for now, Ignite doesn't have "tool strategy" I think
> >> it's
> >>> a big issue from the user's point of view.
> >>> I absolutely agree with that.
> >>>
> >>>> We should solve it in the nearest time. Feel free to start this
> >> activity
> >>> I have no plan at the moment. However, at the first stage, we could
> >>> understand the difference between visor and control utility.
> >>> All useful features from visor should be moved/implemented in control
> >>> utility and after that visor tool and should be marked as
> >>> deprecated/obsoleted.
> >>>
> >>>> You need to throw in control.sh also, which does some kind of
> >> statistics
> >>> too, such as idle_verify.
> >>>> Please, clarify your idea:
> >>>>   We should use some info from control.sh to the report?
> >>>>   The report should be generated by some control.sh subcommand?
> >>> If I am not mistaken, the oracle database has AWR tool (mentioned on
> the
> >>> IEP page) which is a command-line utility that generates HTML reports.
> >>> I like this idea and I think this is a good approach that can be
> realized
> >>> in the control utility.
> >>> If we have a case that cannot be implemented in this way, we have to
> >>> clearly states the difference between these tools so as not to confuse
> >> our
> >>> users.
> >>> What do you think?
> >>>
> >>> Thanks,
> >>> Slava.
> >>>
> >>>
> >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <nizhikov@apache.org>:
> >>>
> >>>> Hello, Slava, Ilya, Denis.
> >>>>
> >>>> Thanks for joining this discussion!
> >>>>
> >>>>> - visor (which is deprecated)
> >>>>
> >>>> Who deprecated visor and when?
> >>>> Maybe I miss something?
> >>>>
> >>>>> - web-console (to be honest, I don't quite understand the status
of
> >> this
> >>>> tool)
> >>>>
> >>>> +1.
> >>>>
> >>>>> I am not against the new tool, I just want to understand the
> >> motivation
> >>>> to not improve the existing sub-projects.
> >>>>
> >>>> It's true that, for now, Ignite doesn't have "tool strategy"
> >>>> I think it's a big issue from the user's point of view.
> >>>> We should solve it in the nearest time.
> >>>> Feel free to start this activity.
> >>>>
> >>>>> - new ignite-profiling (which is a monitoring tool as well, judging
> >> by
> >>>> the provided link [1] )
> >>>>
> >>>> The general idea is the following:
> >>>>
> >>>> 1. We should have some profiling mechanism that will generate a
> >> node-local
> >>>> event log
> >>>> 2. We should have a tool that can export events to some third-party
> >>>> system. This system can be an Elastic Search(Kibana) or Ignite
> >> performance
> >>>> report or Kafka log, whatever.
> >>>> 3. Ignite performance report, in the first release, should be a
> >> "static"
> >>>> tool.
> >>>>    This means we take static logs(that is not rewritten in the
> >> analysis
> >>>> time) and feed them in the script(or tool or control.sh, whatever)
> >>>>    The script produces static report that can be used for overall
> >>>> performance analysis.
> >>>>
> >>>> The primary users of this report is a developer of Ignite based
> >>>> applications and performance engineers.
> >>>>
> >>>> Ilya,
> >>>>
> >>>>> You need to throw in control.sh also, which does some kind of
> >> statistics
> >>>> too, such as idle_verify.
> >>>>
> >>>> Please, clarify your idea:
> >>>>    We should use some info from control.sh to the report?
> >>>>    The report should be generated by some control.sh subcommand?
> >>>>
> >>>>
> >>>> Denis,
> >>>>
> >>>>> Speaking of the probes/statistics collection approach, is it
> >> supposed to
> >>>> reuse tracing capabilities that are to be added as part of IEP-35?
> >>>>
> >>>> For now, we don't have any results of tracing development available
in
> >>>> Apache Ignite.
> >>>> Hopefully, we got some in a couple of weeks.
> >>>> After it, we can start a discussion of how to merge two improvements.
> >>>>
> >>>>
> >>>>
> >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <dmagda@apache.org>
> >> написал(а):
> >>>>>
> >>>>>>
> >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> >> I'm for
> >>>>>> reuse.
> >>>>>
> >>>>>
> >>>>> Looks like we need to sync up on these activities/initiatives to
> >> ensure
> >>>> we
> >>>>> don't do a duplicate job. If you think a separate discussion is
> >> necessary
> >>>>> let's kick it off.
> >>>>>
> >>>>> -
> >>>>> Denis
> >>>>>
> >>>>>
> >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
> >> nsamelchev@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Denis, Ilya,
> >>>>>>
> >>>>>> I will try to integrate profiling functionality into control.sh
> >> utility.
> >>>>>>
> >>>>>>> Speaking of the probes/statistics collection approach, is
it
> >> supposed
> >>>> to
> >>>>>>> reuse tracing capabilities that are to be added as part
of IEP-35?
> >>>>>> Tracing is more deeply takes statistics. If it will be possible,
> >> I'm for
> >>>>>> reuse.
> >>>>>>
> >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <
> >> ilya.kasnacheev@gmail.com
> >>>>> :
> >>>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> I suggest that it's one of the places where it could be
put
> >> instead of
> >>>>>>> adding a new tool.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> --
> >>>>>>> Ilya Kasnacheev
> >>>>>>>
> >>>>>>>
> >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <nsamelchev@gmail.com
> >>> :
> >>>>>>>
> >>>>>>>> Ilya,
> >>>>>>>>
> >>>>>>>> You suggest using control.sh to build the report?
> >>>>>>>>
> >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev
<
> >>>>>> ilya.kasnacheev@gmail.com>:
> >>>>>>>>>
> >>>>>>>>> Hello!
> >>>>>>>>>
> >>>>>>>>> You need to throw in control.sh also, which does
some kind of
> >>>>>> statistics
> >>>>>>>>> too, such as idle_verify.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> --
> >>>>>>>>> Ilya Kasnacheev
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав
Коптилин <
> >>>>>> slava.koptilin@gmail.com
> >>>>>>>>> :
> >>>>>>>>>
> >>>>>>>>>> Hello Nikita,
> >>>>>>>>>>
> >>>>>>>>>> Perhaps, I am missing something...
> >>>>>>>>>> Apache Ignite already has a web-console tool.
Do we want to
> >>>>>> improve the
> >>>>>>>>>> existing tool instead of creating a new one?
> >>>>>>>>>> It seems to me, this can be confusing for users.
> >>>>>>>>>> - visor (which is deprecated)
> >>>>>>>>>> - web-console (to be honest, I don't quite understand
the status
> >>>>>> of
> >>>>>>>> this
> >>>>>>>>>> tool)
> >>>>>>>>>> - new ignite-profiling (which is a monitoring
tool as well,
> >>>>>> judging
> >>>>>>>> by the
> >>>>>>>>>> provided link [1] )
> >>>>>>>>>>
> >>>>>>>>>> I am not against the new tool, I just want to
understand the
> >>>>>>>> motivation to
> >>>>>>>>>> not improve the existing sub-projects.
> >>>>>>>>>>
> >>>>>>>>>> [1]
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> S.
> >>>>>>>>>>
> >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev
<
> >> nsamelchev@gmail.com
> >>>>>>> :
> >>>>>>>>>>
> >>>>>>>>>>> Hi, Igniters.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm working on cluster profiling and the
tool for creating a
> >>>>>>>>>>> performance report. [1] I have prepared
PoC based on
> >> performance
> >>>>>>>>>>> logging to a separate category of Ignite
log. The report
> >>>>>> contains:
> >>>>>>>>>>>
> >>>>>>>>>>> - Cache operations and its distribution
by types [2]
> >>>>>>>>>>> - Transactions and histogram of durations
[3]
> >>>>>>>>>>> - SQL and Scan query statistics, top of
slowest queries,
> >> logical
> >>>>>> and
> >>>>>>>>>>> physical reads by query [4]
> >>>>>>>>>>> - Compute statistics, top of slowest tasks
and their jobs [5]
> >>>>>>>>>>> Soon I will add:
> >>>>>>>>>>> - Topology and Ignite versions info
> >>>>>>>>>>> - Client ID in case of operations from clients
> >>>>>>>>>>>
> >>>>>>>>>>> For now, I'm developing a binary logging
format to reduce the
> >>>>>> effect
> >>>>>>>>>>> on performance. I'll try to reuse Ignite
mechanisms.
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to hear suggestions for the
profiling and the
> >>>>>>>> performance
> >>>>>>>>>>> report.
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >>>>>>>>>>> [2]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> >>>>>>>>>>> [3]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> >>>>>>>>>>> [4]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> >>>>>>>>>>> [5]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Best wishes,
> >>>>>>>>>>> Amelchev Nikita
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Best wishes,
> >>>>>>>> Amelchev Nikita
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best wishes,
> >>>>>> Amelchev Nikita
> >>>>>>
> >>>>
> >>>>
> >>
> >>
> >>
> >> --
> >> Best wishes,
> >> Amelchev Nikita
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message