ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Tue, 03 Nov 2020 12:30:26 GMT
Ksenia, thanks for scheduling this on such short notice!

As for the original topic, I do support Alexey's idea. We're not going to
rewrite anything from scratch, as most of the components are going to be
moved as-is or with minimal modifications. However, the changes that are
proposed imply serious rework of the core parts of the code, which are not
properly decoupled from each other and from other parts. This makes the
incremental approach borderline impossible. Developing in a new repo,
however, addresses this concern. As a bonus, we can also refactor the code,
introduce better decoupling, get rid of kernel context, and develop unit
tests (finally!).

Basically, this proposal only affects the *process*, not the set of changes
we had discussed before. Ignite 3.0 is our unique chance to make things
right.

-Val

On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova <romanova.ks.spb@gmail.com>
wrote:

> Pavel, all the interesting points will be anyway published here in English
> (as the principal "if it's not on devlist it doesn't happened" is still
> relevant). This is just a quick call for a group of developers. Later we
> can do a separate presentation of idea and discussion in English as we did
> for the Ignite 3.0 draft of changes.
>
> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org>:
>
> > Kseniya,
> >
> > Thanks for scheduling this call.
> > Do you think we can switch to English if non-Russian speaking community
> > members decide to join?
> >
> > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <
> romanova.ks.spb@gmail.com
> > >
> > wrote:
> >
> > > Let's do this community discussion open. Here's the link on zoom call
> in
> > > Russian for Friday 6 PM:
> > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> > >
> > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhikov@apache.org>:
> > >
> > > > Time works for me.
> > > >
> > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> > alexey.goncharuk@gmail.com
> > > >
> > > > написал(а):
> > > > >
> > > > > Nikolay,
> > > > >
> > > > > I am up for the call. I will try to explain my reasoning in greater
> > > > detail
> > > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> > work?
> > > > >
> > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhikov@apache.org
> >:
> > > > >
> > > > >> Igniters, should we have a call for this topic?
> > > > >>
> > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org>
> > > > >> написал(а):
> > > > >>>
> > > > >>>> not intend to rewrite everything from scratch
> > > > >>>
> > > > >>>> Every single test from Ignite 2.x should be moved to
Ignite 3
> > > > >>>> regardless of how we choose to proceed.
> > > > >>>
> > > > >>> Alexey, thank you for the explanation, this addresses all
of my
> > > > concerns.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > > >> andrey.mashenkov@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi, Igniters.
> > > > >>>>
> > > > >>>> * AFAIU, we need a new repo if we want to apply different
> > > restrictions
> > > > >> to
> > > > >>>> pull requests,
> > > > >>>> otherwise I see no difference for myself.
> > > > >>>> E.g. make static analysis (do we have?), compile, styles,
and
> > > javadoc
> > > > >>>> checks mandatory.
> > > > >>>>
> > > > >>>> I think that relaxed requirements here will lead to bad
product
> > > > quality.
> > > > >>>>
> > > > >>>> * Agree with Pavel, we should 'keep' integrations tests
somehow.
> > > > >>>> During active development tests will be broken most of
time, so,
> > > > >>>> I'd port them e.g. suite-by-suite once we will have a
stable and
> > > > >> featured
> > > > >>>> environment to run them and of course make test's code
clear and
> > > avoid
> > > > >>>> bad/non-relevant ones.
> > > > >>>>
> > > > >>>> * I like bottom-up approach.
> > > > >>>> With it we could make a better framework. I mean clear
component
> > > > >> lifecycle,
> > > > >>>> component wiring mechanics, general methods to approach
core
> > > > components
> > > > >>>> such as exchange/communication
> > > > >>>> to avoid code mess like we have in ExchangeFuture with
all these
> > > > custom
> > > > >>>> callbacks for each component, interfaces like
> > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport
and
> > > > >>>> a pack of
> > > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > > >>>> and so on in various unexpected places.
> > > > >>>> Hope, we will be able to port most of the good code to
the new
> > > > framework
> > > > >>>> version.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > > >>>> alexey.goncharuk@gmail.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Nikolay, Pavel,
> > > > >>>>>
> > > > >>>>> Thanks for the feedback! First of all, I wanted to
stress that
> I
> > do
> > > > not
> > > > >>>>> intend to rewrite everything from scratch (I never
used this
> > > phrase).
> > > > >>>> There
> > > > >>>>> are significant parts of code that would be moved
with minimal
> > > > >>>>> modifications. Second, I never said that we will
get rid of the
> > old
> > > > >> tests
> > > > >>>>> codebase. Every single test from Ignite 2.x should
be moved to
> > > > Ignite 3
> > > > >>>>> regardless of how we choose to proceed.
> > > > >>>>>
> > > > >>>>> My point is that for some parts of the code a clean
bottom-up
> > > > >>>>> implementation will be cheaper in many ways. Let
me give you a
> > few
> > > > >>>> concrete
> > > > >>>>> examples:
> > > > >>>>>
> > > > >>>>>  - I think no one can object that we need a cleanly
separated
> > > > >>>> persistence
> > > > >>>>>  layer for Ignite. There is a very raw draft IEP
for this
> > already.
> > > On
> > > > >>>> the
> > > > >>>>>  other hand, I think we also can agree that we need
a
> split-brain
> > > > >>>>> resistant
> > > > >>>>>  replication protocol for caches. There is also an
IEP for
> this.
> > > > >>>> Neither
> > > > >>>>> of
> > > > >>>>>  the changes is a good fit for 2.x because they are
likely to
> > > > >> introduce
> > > > >>>>>  breaking changes in the persistence layer, configuration
and
> > > > >> behavior.
> > > > >>>>>  Additionally, these components are now tightly coupled,
so
> there
> > > is
> > > > >> no
> > > > >>>>> way
> > > > >>>>>  these two changes can be implemented in parallel
and then
> merged
> > > > >>>>> together
> > > > >>>>>  easily. So what we will end up with is having to
implement
> these
> > > > >>>> changes
> > > > >>>>>  sequentially, fixing all existing tests twice, and
essentially
> > > > >>>> throwing
> > > > >>>>>  away half of the work done because the other part
of the
> change
> > is
> > > > >>>>>  re-implemented
> > > > >>>>>  - Similar example goes with getting rid of
> IgniteInternalFuture
> > > and
> > > > >>>>>  replacing it with CompletableFuture, and any other
change that
> > > > >> touches
> > > > >>>>> the
> > > > >>>>>  asynchronous part of the code.
> > > > >>>>>
> > > > >>>>> Third, I do not see how this choice affects the UX
of Ignite.
> The
> > > end
> > > > >>>> user
> > > > >>>>> experience must be fixed in the IEP regardless of
the
> development
> > > > >> process
> > > > >>>>> and the fact that we have gaps in this area in Ignite
2.x just
> > > > confirms
> > > > >>>>> that.
> > > > >>>>>
> > > > >>>>> Pavel, agree that a repo/branch is a technicality,
I guess if
> > > > >>>> reformulate,
> > > > >>>>> my point is that we might agree to have a single
development
> > master
> > > > >>>> branch
> > > > >>>>> with 'disassembled' end-to-end functionality for
some period of
> > > time
> > > > to
> > > > >>>>> speed up development, and re-assemble the core features
after
> > > having
> > > > >>>>> submodules tested independently.
> > > > >>>>>
> > > > >>>>> Nikolay,
> > > > >>>>>> We have many features that have to evolve.
> > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
support,
> etc.
> > > > >>>>> This is not very specific. In the end, resources
are limited
> and
> > we
> > > > >> will
> > > > >>>>> not be able to drive both tracks simultaneously,
especially
> > after a
> > > > >>>> couple
> > > > >>>>> of features having been implemented for Ignite 3.0.
If there
> are
> > > > indeed
> > > > >>>>> some major changes that we want to do in Ignite 2.x
instead of
> > > > putting
> > > > >>>>> effort into 3.0 - let's discuss them. I am just not
aware of
> any,
> > > > >> that's
> > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > >>>>>
> > > > >>>>>> We have bugs and issues that can be fixed in
2.x without
> > breaking
> > > > >>>> backward
> > > > >>>>> compatibility.
> > > > >>>>>> We have many users who are happy with the 2.x
with all it’s
> > > issues.
> > > > >>>>> These changes will be covered by end-to-end tests
and migrated
> to
> > > > >> Ignite
> > > > >>>>> 3.0, so I see no issues here.
> > > > >>>>>
> > > > >>>>> Finally, Anton & Nikolay
> > > > >>>>> I do not have an estimate for this simply because
the activity
> is
> > > > >>>>> community-driven and it depends on the number of
people willing
> > to
> > > > >>>>> contribute. With the current pace, I would hope to
have an RC
> of
> > > > Ignite
> > > > >>>> 3.0
> > > > >>>>> to be ready by the end of 2021. My gut feeling is
that by
> moving
> > > with
> > > > >>>>> incremental changes, we will not be able to implement
even half
> > of
> > > > the
> > > > >>>>> wishlist by that time.
> > > > >>>>> I doubt that releasing several major releases with
breaking
> > changes
> > > > >> will
> > > > >>>>> make Ignite users happy either because each upgrade
will cost
> > > Ignite
> > > > >>>> users
> > > > >>>>> money, so the fewer major versions we release, the
better. Thus
> > my
> > > > wish
> > > > >>>> to
> > > > >>>>> include all breaking changes in one release.
> > > > >>>>>
> > > > >>>>> I'll be now quiet for a while, let's see what other
community
> > > members
> > > > >>>>> think.
> > > > >>>>>
> > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn
<
> > ptupitsyn@apache.org
> > > >:
> > > > >>>>>
> > > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > > >>>>>> We don't want to follow the path of Netscape
and lose all our
> > > users
> > > > >>>>>> by the time we have a working 3.0 [1]
> > > > >>>>>>
> > > > >>>>>> 2. Not sure about new repo - seems like some
pain and no gain,
> > > > what's
> > > > >>>> the
> > > > >>>>>> problem with a branch?
> > > > >>>>>>
> > > > >>>>>> 3. We should keep existing integration tests
when possible.
> > > > >>>>>> We have accumulated a lot of edge case knowledge
over the
> years,
> > > > >>>>>> it is not a good idea to send all of that down
the drain.
> > > > >>>>>> Yes, integration tests are slow, but they are
the most
> valuable.
> > > > >>>>>> I think we can move more stuff into nightly runs
and have a
> fast
> > > and
> > > > >>>>> modern
> > > > >>>>>> basic suite.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Alexey, you are much more familiar with the Ignite
core
> codebase
> > > > than
> > > > >>>>> most
> > > > >>>>>> of us,
> > > > >>>>>> can you please explain in more detail which particular
> feature,
> > in
> > > > >> your
> > > > >>>>>> opinion,
> > > > >>>>>> mandates this "start from scratch" approach?
> > > > >>>>>> Is it really not possible at all to follow a
less radical way?
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> [1]
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > > >>>>>>
> > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov
<
> > > nizhikov@apache.org
> > > > >
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hello, Alexey.
> > > > >>>>>>>
> > > > >>>>>>> I think that «rewriting from scratch» approach
has a high
> risk
> > to
> > > > >>>> make
> > > > >>>>>> new
> > > > >>>>>>> features unusable.
> > > > >>>>>>> At the time Ignite2 was started no-one wants
to do bad UX or
> > bad
> > > > >>>>>> features.
> > > > >>>>>>> Nevertheless, it happen.
> > > > >>>>>>>
> > > > >>>>>>> I think we can avoid it with the Ignite3
and successors if we
> > > will
> > > > >>>> move
> > > > >>>>>>> step by step without keeping backward compatibility
> > > > >>>>>>> With the step by step approach, we can focus
on each
> component
> > > > >>>>>> separately.
> > > > >>>>>>>
> > > > >>>>>>>> What new features are we planning to
implement for Ignite
> 2.x?
> > > > >>>>>>>
> > > > >>>>>>> We have many features that have to evolve.
> > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
support,
> etc.
> > > > >>>>>>> We have bugs and issues that can be fixed
in 2.x without
> > breaking
> > > > >>>>>> backward
> > > > >>>>>>> compatibility.
> > > > >>>>>>> We have many users who are happy with the
2.x with all it’s
> > > issues.
> > > > >>>>>>>
> > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton
Vinogradov <av@apache.org>
> > > > >>>>> написал(а):
> > > > >>>>>>>>
> > > > >>>>>>>> Alexey,
> > > > >>>>>>>>
> > > > >>>>>>>> Do we have any estimates of how fast
we'll be able to gain
> > > > >>>>>>> production-ready
> > > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > > >>>>>>>>
> > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey
Goncharuk <
> > > > >>>>>>> alexey.goncharuk@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Nikolay,
> > > > >>>>>>>>>
> > > > >>>>>>>>> What new features are we planning
to implement for Ignite
> > 2.x?
> > > I
> > > > >>>>> think
> > > > >>>>>>> once
> > > > >>>>>>>>> we commence working on Ignite 3.0,
we should gradually
> cease
> > > the
> > > > >>>>>>> activity
> > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because
such parallel
> > > development
> > > > >>>>> will
> > > > >>>>>> be
> > > > >>>>>>>>> overwhelming regardless of how we
choose to proceed.
> > > > >>>>>>>>>
> > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38,
Nikolay Izhikov <
> > > > nizhikov@apache.org
> > > > >>>>> :
> > > > >>>>>>>>>
> > > > >>>>>>>>>> To be clear:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> I would suggest creating
a new repository for Ignite 3.0
> > > > >>>>> (perhaps, a
> > > > >>>>>>>>> new
> > > > >>>>>>>>>> clean branch, but a new repo
looks nicer to me) and a new
> > > Ignite
> > > > >>>>> 3.0
> > > > >>>>>>>>>> TeamCity project.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> +1 for new Team City project.
> > > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > > >>>>>>>>>> -1 for new repo.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> 2 нояб. 2020 г., в
13:35, Nikolay Izhikov <
> > > > >>>> NIzhikov.dev@gmail.com
> > > > >>>>>>
> > > > >>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Hello, Alexey.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I think it will hurt our
project more than help.
> > > > >>>>>>>>>>> Developing new features for
2 separate branches with the
> > > > >>>> different
> > > > >>>>>>> APIs
> > > > >>>>>>>>>> and internal structure is overwhelming
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Maybe we should relax a bit
requirements for Ignite3?
> > > > >>>>>>>>>>> Maybe we should move step
by step and make Ignite3 with
> new
> > > > >>>>>>>>>> configuration than Ignite4 with
new transactions, etc?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> 2 нояб. 2020 г.,
в 13:14, Alexey Goncharuk <
> > > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > > >>>>>>>>>> написал(а):
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Igniters,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I wanted to pitch a rather
radical idea regarding the
> > Ignite
> > > > >>>> 3.0
> > > > >>>>>>>>>>>> development which has
occurred to me some time ago.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> We already have several
IEPs targeted to Ignite 3.0
> which
> > > > imply
> > > > >>>>>> major
> > > > >>>>>>>>>>>> changes to the codebase
(the change in replication
> > protocol
> > > > and
> > > > >>>>>> thus
> > > > >>>>>>>>>>>> transactions, change
in binary format, updated
> > metastorage,
> > > > >>>> etc).
> > > > >>>>>> We
> > > > >>>>>>>>>> also
> > > > >>>>>>>>>>>> planned significant changes
in public APIs:
> configuration
> > > > >>>> format
> > > > >>>>>>>>> change,
> > > > >>>>>>>>>>>> improvements in cache
APIs, SQL APIs, transaction mode
> > > rework.
> > > > >>>>> The
> > > > >>>>>>>>>> wishlist
> > > > >>>>>>>>>>>> of changes for Ignite
3.0 is huge.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> So, I was wondering whether
it makes sense to try to
> > change
> > > > the
> > > > >>>>> old
> > > > >>>>>>>>>>>> codebase, or start with
a new baseline and move old
> pieces
> > > of
> > > > >>>>> code
> > > > >>>>>>>>> that
> > > > >>>>>>>>>> do
> > > > >>>>>>>>>>>> not require significant
rework. Personally, I would go
> > with
> > > > the
> > > > >>>>>>> second
> > > > >>>>>>>>>>>> option for the following
reasons:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> - We have a chance to
shift the development paradigm in
> > the
> > > > >>>>> project
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>> introduce the practice
of true unit-tests. In the new
> > > baseline
> > > > >>>> at
> > > > >>>>>> the
> > > > >>>>>>>>>>>> beginning there will
be no ability to run an end-to-end
> > > > >>>> scenario,
> > > > >>>>>>>>> thus
> > > > >>>>>>>>>> we
> > > > >>>>>>>>>>>> will be forced to write
unit-tests. So far, such
> practice
> > > was
> > > > >>>>> hard
> > > > >>>>>> to
> > > > >>>>>>>>>>>> implement because of
tight coupling between Ignite
> > > components
> > > > >>>> and
> > > > >>>>>>>>>> inability
> > > > >>>>>>>>>>>> to instantiate components
without an instance of
> > > > KernalContext.
> > > > >>>>> For
> > > > >>>>>>>>>>>> example, we should be
able to thoroughly test internal
> > > > >>>>> primitives,
> > > > >>>>>>>>>> such as
> > > > >>>>>>>>>>>> replication protocol
(without actual communication),
> > > > >>>> distributed
> > > > >>>>>>>>>>>> metastorage contracts,
persistence layer, etc.
> > > > >>>>>>>>>>>> - We will significantly
reduce the development cycle in
> > the
> > > > >>>>>> beginning
> > > > >>>>>>>>>>>> (right now the RunAll
takes two hours of astronomical
> time
> > > > with
> > > > >>>>>> empty
> > > > >>>>>>>>>> TC;
> > > > >>>>>>>>>>>> in the new approach developer
will be able to run ALL
> > tests
> > > > >>>>> locally
> > > > >>>>>>>>> in
> > > > >>>>>>>>>> a
> > > > >>>>>>>>>>>> matter of minutes)
> > > > >>>>>>>>>>>> - We can get rid of TC
bot and enforce green TC by
> > > integrating
> > > > >>>> TC
> > > > >>>>>>>>> build
> > > > >>>>>>>>>>>> results with GitHub PRs
(the same way Travis is
> currently
> > > > >>>>>> integrated
> > > > >>>>>>>>>> to PR
> > > > >>>>>>>>>>>> check). We should restrict
PR merge without a TC check
> > > > >>>>>>>>>>>> - We will still have
to re-write all tests, but only
> once.
> > > If
> > > > >>>> we
> > > > >>>>>> try
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>> modify the old codebase,
we would need to modify all the
> > > tests
> > > > >>>>> for
> > > > >>>>>>>>>> every
> > > > >>>>>>>>>>>> major change (public
API change, configuration change)
> > > > >>>>>>>>>>>> - We will have fewer
conflicts when working together.
> For
> > > > >>>>> example,
> > > > >>>>>> I
> > > > >>>>>>>>>>>> cannot imagine how one
would merge two changes of
> getting
> > > rid
> > > > >>>> of
> > > > >>>>>>>>>>>> IgniteFuture and changes
in replication protocol, for
> > > example
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Technically, I would
suggest creating a new repository
> for
> > > > >>>> Ignite
> > > > >>>>>> 3.0
> > > > >>>>>>>>>>>> (perhaps, a new clean
branch, but a new repo looks nicer
> > to
> > > > me)
> > > > >>>>>> and a
> > > > >>>>>>>>>> new
> > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> While it may seem quite
radical, I do believe that this
> > > > >>>> approach
> > > > >>>>>> will
> > > > >>>>>>>>>> give
> > > > >>>>>>>>>>>> us more benefits than
trying to make such major changes
> in
> > > the
> > > > >>>>>>>>> existing
> > > > >>>>>>>>>>>> codebase. If needed,
let's schedule a discord chat like
> > > before
> > > > >>>> to
> > > > >>>>>>>>>> discuss
> > > > >>>>>>>>>>>> this.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Best regards,
> > > > >>>> Andrey V. Mashenkov
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message