ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: Tool for performance statistics reports
Date Sun, 21 Jun 2020 20:34:09 GMT
Hi Nikita,

I have reviewed the PR and shared comments. I also had a question on
including files with a MIT license.

Are we ok to use both Apache and MIT licenses in our source files?

Regards,
Saikat

On Fri, Jun 12, 2020 at 8:45 PM Saikat Maitra <saikat.maitra@gmail.com>
wrote:

> Hello Nikita,
>
> Thank you for sharing the information, it is very helpful.
>
> Regards,
> Saikat
>
> On Thu, Jun 11, 2020 at 3:20 AM Nikita Amelchev <nsamelchev@gmail.com>
> wrote:
>
>> Hello,
>>
>> > Can you please share more info on how we can use the profiling tool with
>> > ignite-extensions modules?
>>
>> I have updated the module readme file. You can find instructions
>> there. For now, I am working on review fixes and marked PR as a draft.
>>
>> чт, 11 июн. 2020 г. в 03:21, Saikat Maitra <saikat.maitra@gmail.com>:
>> >
>> > Hello Nikita,
>> >
>> > I observed we have an open PR in ignite repo for this feature with
>> > different set of changes compared to ignite extensions repo.
>> >
>> >
>> > apache/ignite#7693 <https://github.com/apache/ignite/pull/7693>
>> >
>> > https://github.com/apache/ignite-extensions/pull/16
>> >
>> > Can you please share more info on how we can use the profiling tool with
>> > ignite-extensions modules?
>> >
>> >
>> > Regards,
>> >
>> > Saikat
>> >
>> > On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov <nizhikov@apache.org>
>> wrote:
>> >
>> > > Hello, Alexey.
>> > >
>> > > Thanks for the review.
>> > >
>> > > My understanding if the following:
>> > >
>> > > We will have 3 in-depth tool to find issues in cluster:
>> > >
>> > > 1. Metrics + System views - data that describe Ignite entities very
>> > > high-level.
>> > >
>> > > 2. Profiling - tool to know what specific query of transactions are
>> slow.
>> > > In many cases, this knowledge is enough to fix the issue(rewrite
>> query,
>> > > redesign transactions flow, etc)
>> > >
>> > > 3. Tracing - tool to know why one of 1000 of the same queries was
>> slow.
>> > > The most detailed view of the Ignite internal processes.
>> > >
>> > > > For example, a user would not be able to match a long task with a
>> long
>> > > job in that task.
>> > >
>> > > This is not true.
>> > > Profiling report will aggregate data from all nodes.
>> > > So there will be both
>> > >
>> > >  * summary time of the task
>> > >  * time of the each job in the task.
>> > >
>> > >
>> > > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <
>> alexey.goncharuk@gmail.com>
>> > > написал(а):
>> > > >
>> > > > Nikita, Igniters,
>> > > >
>> > > > I left a few comments on the tool itself in the PR.
>> > > >
>> > > > However, I would like to reiterate and discuss why a user would
>> prefer to
>> > > > use the profiling tool over tracing? Profiling tool only captures
>> very
>> > > > high-level details of the operations (a single cache operation, for
>> > > > example), and does not interconnect operations happened on different
>> > > nodes.
>> > > > For example, a user would not be able to match a long task with a
>> long
>> > > job
>> > > > in that task. In other words, profiling data is always a subset of
>> data
>> > > > collected by tracing.
>> > > >
>> > > > Maybe it makes sense to adopt local log file approach to write
>> spans so
>> > > we
>> > > > can process those spans later to build a report?
>> > > >
>> > > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev <nsamelchev@gmail.com>:
>> > > >
>> > > >> Hi, Igniters.
>> > > >>
>> > > >> I have implemented cluster profiling and tool to build the
>> performance
>> > > >> report. It's ready to be reviewed. [1, 2]
>> > > >>
>> > > >> Profiling can be managed by JMX bean. I have plans to implement
it
>> to
>> > > >> control.sh also.
>> > > >>
>> > > >> Nodes write statistics to the temporary off heap buffer and then
>> one
>> > > >> thread flushes to the profiling files. The write mechanics and
>> format
>> > > >> is like WAL logging.
>> > > >> The report contains the following statistics:
>> > > >> - nodes and caches info
>> > > >> - cache operations and transaction statistics
>> > > >> - SQL and scan queries statistics (include logical and physical
>> reads
>> > > per
>> > > >> query)
>> > > >> - tasks and jobs statistics.
>> > > >>
>> > > >> More details in the IEP [3].
>> > > >>
>> > > >> [1] https://github.com/apache/ignite/pull/7693
>> > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
>> > > >> [3]
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>> > > >>
>> > > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин
<
>> > > slava.koptilin@gmail.com>:
>> > > >>>
>> > > >>> Hello Nikolay,
>> > > >>>
>> > > >>>> Who deprecated visor and when? Maybe I miss something?
>> > > >>> On the one hand, there was technically no community consensus
>> that this
>> > > >>> tool should be obsolete.
>> > > >>> On the other hand, my opinion based on the following topic:
>> > > >>>
>> > > >>
>> > >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Re-Visor-plugin-tp44879p44939.html
>> > > >>> Moreover, it seems to me, currently, the control utility is
>> widely used
>> > > >> and
>> > > >>> actively developed, instead of the visor.
>> > > >>>
>> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
I
>> think
>> > > >> it's
>> > > >>> a big issue from the user's point of view.
>> > > >>> I absolutely agree with that.
>> > > >>>
>> > > >>>> We should solve it in the nearest time. Feel free to start
this
>> > > >> activity
>> > > >>> I have no plan at the moment. However, at the first stage,
we
>> could
>> > > >>> understand the difference between visor and control utility.
>> > > >>> All useful features from visor should be moved/implemented
in
>> control
>> > > >>> utility and after that visor tool and should be marked as
>> > > >>> deprecated/obsoleted.
>> > > >>>
>> > > >>>> You need to throw in control.sh also, which does some
kind of
>> > > >> statistics
>> > > >>> too, such as idle_verify.
>> > > >>>> Please, clarify your idea:
>> > > >>>>   We should use some info from control.sh to the report?
>> > > >>>>   The report should be generated by some control.sh subcommand?
>> > > >>> If I am not mistaken, the oracle database has AWR tool (mentioned
>> on
>> > > the
>> > > >>> IEP page) which is a command-line utility that generates HTML
>> reports.
>> > > >>> I like this idea and I think this is a good approach that
can be
>> > > realized
>> > > >>> in the control utility.
>> > > >>> If we have a case that cannot be implemented in this way,
we have
>> to
>> > > >>> clearly states the difference between these tools so as not
to
>> confuse
>> > > >> our
>> > > >>> users.
>> > > >>> What do you think?
>> > > >>>
>> > > >>> Thanks,
>> > > >>> Slava.
>> > > >>>
>> > > >>>
>> > > >>> сб, 25 апр. 2020 г. в 12:00, Nikolay Izhikov <nizhikov@apache.org
>> >:
>> > > >>>
>> > > >>>> Hello, Slava, Ilya, Denis.
>> > > >>>>
>> > > >>>> Thanks for joining this discussion!
>> > > >>>>
>> > > >>>>> - visor (which is deprecated)
>> > > >>>>
>> > > >>>> Who deprecated visor and when?
>> > > >>>> Maybe I miss something?
>> > > >>>>
>> > > >>>>> - web-console (to be honest, I don't quite understand
the
>> status of
>> > > >> this
>> > > >>>> tool)
>> > > >>>>
>> > > >>>> +1.
>> > > >>>>
>> > > >>>>> I am not against the new tool, I just want to understand
the
>> > > >> motivation
>> > > >>>> to not improve the existing sub-projects.
>> > > >>>>
>> > > >>>> It's true that, for now, Ignite doesn't have "tool strategy"
>> > > >>>> I think it's a big issue from the user's point of view.
>> > > >>>> We should solve it in the nearest time.
>> > > >>>> Feel free to start this activity.
>> > > >>>>
>> > > >>>>> - new ignite-profiling (which is a monitoring tool
as well,
>> judging
>> > > >> by
>> > > >>>> the provided link [1] )
>> > > >>>>
>> > > >>>> The general idea is the following:
>> > > >>>>
>> > > >>>> 1. We should have some profiling mechanism that will generate
a
>> > > >> node-local
>> > > >>>> event log
>> > > >>>> 2. We should have a tool that can export events to some
>> third-party
>> > > >>>> system. This system can be an Elastic Search(Kibana) or
Ignite
>> > > >> performance
>> > > >>>> report or Kafka log, whatever.
>> > > >>>> 3. Ignite performance report, in the first release, should
be a
>> > > >> "static"
>> > > >>>> tool.
>> > > >>>>    This means we take static logs(that is not rewritten
in the
>> > > >> analysis
>> > > >>>> time) and feed them in the script(or tool or control.sh,
>> whatever)
>> > > >>>>    The script produces static report that can be used
for overall
>> > > >>>> performance analysis.
>> > > >>>>
>> > > >>>> The primary users of this report is a developer of Ignite
based
>> > > >>>> applications and performance engineers.
>> > > >>>>
>> > > >>>> Ilya,
>> > > >>>>
>> > > >>>>> You need to throw in control.sh also, which does some
kind of
>> > > >> statistics
>> > > >>>> too, such as idle_verify.
>> > > >>>>
>> > > >>>> Please, clarify your idea:
>> > > >>>>    We should use some info from control.sh to the report?
>> > > >>>>    The report should be generated by some control.sh subcommand?
>> > > >>>>
>> > > >>>>
>> > > >>>> Denis,
>> > > >>>>
>> > > >>>>> Speaking of the probes/statistics collection approach,
is it
>> > > >> supposed to
>> > > >>>> reuse tracing capabilities that are to be added as part
of
>> IEP-35?
>> > > >>>>
>> > > >>>> For now, we don't have any results of tracing development
>> available in
>> > > >>>> Apache Ignite.
>> > > >>>> Hopefully, we got some in a couple of weeks.
>> > > >>>> After it, we can start a discussion of how to merge two
>> improvements.
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>>> 24 апр. 2020 г., в 20:32, Denis Magda <dmagda@apache.org>
>> > > >> написал(а):
>> > > >>>>>
>> > > >>>>>>
>> > > >>>>>> Tracing is more deeply takes statistics. If it
will be
>> possible,
>> > > >> I'm for
>> > > >>>>>> reuse.
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> Looks like we need to sync up on these activities/initiatives
to
>> > > >> ensure
>> > > >>>> we
>> > > >>>>> don't do a duplicate job. If you think a separate
discussion is
>> > > >> necessary
>> > > >>>>> let's kick it off.
>> > > >>>>>
>> > > >>>>> -
>> > > >>>>> Denis
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <
>> > > >> nsamelchev@gmail.com>
>> > > >>>>> wrote:
>> > > >>>>>
>> > > >>>>>> Denis, Ilya,
>> > > >>>>>>
>> > > >>>>>> I will try to integrate profiling functionality
into control.sh
>> > > >> utility.
>> > > >>>>>>
>> > > >>>>>>> Speaking of the probes/statistics collection
approach, is it
>> > > >> supposed
>> > > >>>> to
>> > > >>>>>>> reuse tracing capabilities that are to be
added as part of
>> IEP-35?
>> > > >>>>>> Tracing is more deeply takes statistics. If it
will be
>> possible,
>> > > >> I'm for
>> > > >>>>>> reuse.
>> > > >>>>>>
>> > > >>>>>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev
<
>> > > >> ilya.kasnacheev@gmail.com
>> > > >>>>> :
>> > > >>>>>>>
>> > > >>>>>>> Hello!
>> > > >>>>>>>
>> > > >>>>>>> I suggest that it's one of the places where
it could be put
>> > > >> instead of
>> > > >>>>>>> adding a new tool.
>> > > >>>>>>>
>> > > >>>>>>> Regards,
>> > > >>>>>>> --
>> > > >>>>>>> Ilya Kasnacheev
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> пт, 24 апр. 2020 г. в 18:56, Nikita
Amelchev <
>> nsamelchev@gmail.com
>> > > >>> :
>> > > >>>>>>>
>> > > >>>>>>>> Ilya,
>> > > >>>>>>>>
>> > > >>>>>>>> You suggest using control.sh to build
the report?
>> > > >>>>>>>>
>> > > >>>>>>>> пт, 24 апр. 2020 г. в 18:20, Ilya
Kasnacheev <
>> > > >>>>>> ilya.kasnacheev@gmail.com>:
>> > > >>>>>>>>>
>> > > >>>>>>>>> Hello!
>> > > >>>>>>>>>
>> > > >>>>>>>>> You need to throw in control.sh also,
which does some kind
>> of
>> > > >>>>>> statistics
>> > > >>>>>>>>> too, such as idle_verify.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Regards,
>> > > >>>>>>>>> --
>> > > >>>>>>>>> Ilya Kasnacheev
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> пт, 24 апр. 2020 г. в 18:06,
Вячеслав Коптилин <
>> > > >>>>>> slava.koptilin@gmail.com
>> > > >>>>>>>>> :
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Hello Nikita,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Perhaps, I am missing something...
>> > > >>>>>>>>>> Apache Ignite already has a web-console
tool. Do we want to
>> > > >>>>>> improve the
>> > > >>>>>>>>>> existing tool instead of creating
a new one?
>> > > >>>>>>>>>> It seems to me, this can be confusing
for users.
>> > > >>>>>>>>>> - visor (which is deprecated)
>> > > >>>>>>>>>> - web-console (to be honest, I
don't quite understand the
>> status
>> > > >>>>>> of
>> > > >>>>>>>> this
>> > > >>>>>>>>>> tool)
>> > > >>>>>>>>>> - new ignite-profiling (which
is a monitoring tool as well,
>> > > >>>>>> judging
>> > > >>>>>>>> by the
>> > > >>>>>>>>>> provided link [1] )
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> I am not against the new tool,
I just want to understand
>> the
>> > > >>>>>>>> motivation to
>> > > >>>>>>>>>> not improve the existing sub-projects.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> [1]
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Thanks,
>> > > >>>>>>>>>> S.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> пт, 24 апр. 2020 г. в 14:58,
Nikita Amelchev <
>> > > >> nsamelchev@gmail.com
>> > > >>>>>>> :
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Hi, Igniters.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I'm working on cluster profiling
and the tool for
>> creating a
>> > > >>>>>>>>>>> performance report. [1] I
have prepared PoC based on
>> > > >> performance
>> > > >>>>>>>>>>> logging to a separate category
of Ignite log. The report
>> > > >>>>>> contains:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> - Cache operations and its
distribution by types [2]
>> > > >>>>>>>>>>> - Transactions and histogram
of durations [3]
>> > > >>>>>>>>>>> - SQL and Scan query statistics,
top of slowest queries,
>> > > >> logical
>> > > >>>>>> and
>> > > >>>>>>>>>>> physical reads by query [4]
>> > > >>>>>>>>>>> - Compute statistics, top
of slowest tasks and their jobs
>> [5]
>> > > >>>>>>>>>>> Soon I will add:
>> > > >>>>>>>>>>> - Topology and Ignite versions
info
>> > > >>>>>>>>>>> - Client ID in case of operations
from clients
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> For now, I'm developing a
binary logging format to reduce
>> the
>> > > >>>>>> effect
>> > > >>>>>>>>>>> on performance. I'll try to
reuse Ignite mechanisms.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I would like to hear suggestions
for the profiling and the
>> > > >>>>>>>> performance
>> > > >>>>>>>>>>> report.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> [1]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>> > > >>>>>>>>>>> [2]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
>> > > >>>>>>>>>>> [3]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
>> > > >>>>>>>>>>> [4]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
>> > > >>>>>>>>>>> [5]
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> --
>> > > >>>>>>>>>>> Best wishes,
>> > > >>>>>>>>>>> Amelchev Nikita
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>> --
>> > > >>>>>>>> Best wishes,
>> > > >>>>>>>> Amelchev Nikita
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> --
>> > > >>>>>> Best wishes,
>> > > >>>>>> Amelchev Nikita
>> > > >>>>>>
>> > > >>>>
>> > > >>>>
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Best wishes,
>> > > >> Amelchev Nikita
>> > > >>
>> > >
>> > >
>>
>>
>>
>> --
>> Best wishes,
>> Amelchev Nikita
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message