ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kseniya Romanova <romanova.ks....@gmail.com>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Tue, 03 Nov 2020 11:05:35 GMT
Pavel, all the interesting points will be anyway published here in English
(as the principal "if it's not on devlist it doesn't happened" is still
relevant). This is just a quick call for a group of developers. Later we
can do a separate presentation of idea and discussion in English as we did
for the Ignite 3.0 draft of changes.

вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupitsyn@apache.org>:

> Kseniya,
>
> Thanks for scheduling this call.
> Do you think we can switch to English if non-Russian speaking community
> members decide to join?
>
> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <romanova.ks.spb@gmail.com
> >
> wrote:
>
> > Let's do this community discussion open. Here's the link on zoom call in
> > Russian for Friday 6 PM:
> > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
> >
> > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhikov@apache.org>:
> >
> > > Time works for me.
> > >
> > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <
> alexey.goncharuk@gmail.com
> > >
> > > написал(а):
> > > >
> > > > Nikolay,
> > > >
> > > > I am up for the call. I will try to explain my reasoning in greater
> > > detail
> > > > and will be glad to hear the concerns. Will this Friday, Nov 6th,
> work?
> > > >
> > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhikov@apache.org>:
> > > >
> > > >> Igniters, should we have a call for this topic?
> > > >>
> > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org>
> > > >> написал(а):
> > > >>>
> > > >>>> not intend to rewrite everything from scratch
> > > >>>
> > > >>>> Every single test from Ignite 2.x should be moved to Ignite
3
> > > >>>> regardless of how we choose to proceed.
> > > >>>
> > > >>> Alexey, thank you for the explanation, this addresses all of my
> > > concerns.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > > >> andrey.mashenkov@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi, Igniters.
> > > >>>>
> > > >>>> * AFAIU, we need a new repo if we want to apply different
> > restrictions
> > > >> to
> > > >>>> pull requests,
> > > >>>> otherwise I see no difference for myself.
> > > >>>> E.g. make static analysis (do we have?), compile, styles,
and
> > javadoc
> > > >>>> checks mandatory.
> > > >>>>
> > > >>>> I think that relaxed requirements here will lead to bad product
> > > quality.
> > > >>>>
> > > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > > >>>> During active development tests will be broken most of time,
so,
> > > >>>> I'd port them e.g. suite-by-suite once we will have a stable
and
> > > >> featured
> > > >>>> environment to run them and of course make test's code clear
and
> > avoid
> > > >>>> bad/non-relevant ones.
> > > >>>>
> > > >>>> * I like bottom-up approach.
> > > >>>> With it we could make a better framework. I mean clear component
> > > >> lifecycle,
> > > >>>> component wiring mechanics, general methods to approach core
> > > components
> > > >>>> such as exchange/communication
> > > >>>> to avoid code mess like we have in ExchangeFuture with all
these
> > > custom
> > > >>>> callbacks for each component, interfaces like
> > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > > >>>> a pack of
> > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > > >>>> and so on in various unexpected places.
> > > >>>> Hope, we will be able to port most of the good code to the
new
> > > framework
> > > >>>> version.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > > >>>> alexey.goncharuk@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Nikolay, Pavel,
> > > >>>>>
> > > >>>>> Thanks for the feedback! First of all, I wanted to stress
that I
> do
> > > not
> > > >>>>> intend to rewrite everything from scratch (I never used
this
> > phrase).
> > > >>>> There
> > > >>>>> are significant parts of code that would be moved with
minimal
> > > >>>>> modifications. Second, I never said that we will get rid
of the
> old
> > > >> tests
> > > >>>>> codebase. Every single test from Ignite 2.x should be
moved to
> > > Ignite 3
> > > >>>>> regardless of how we choose to proceed.
> > > >>>>>
> > > >>>>> My point is that for some parts of the code a clean bottom-up
> > > >>>>> implementation will be cheaper in many ways. Let me give
you a
> few
> > > >>>> concrete
> > > >>>>> examples:
> > > >>>>>
> > > >>>>>  - I think no one can object that we need a cleanly separated
> > > >>>> persistence
> > > >>>>>  layer for Ignite. There is a very raw draft IEP for this
> already.
> > On
> > > >>>> the
> > > >>>>>  other hand, I think we also can agree that we need a
split-brain
> > > >>>>> resistant
> > > >>>>>  replication protocol for caches. There is also an IEP
for this.
> > > >>>> Neither
> > > >>>>> of
> > > >>>>>  the changes is a good fit for 2.x because they are likely
to
> > > >> introduce
> > > >>>>>  breaking changes in the persistence layer, configuration
and
> > > >> behavior.
> > > >>>>>  Additionally, these components are now tightly coupled,
so there
> > is
> > > >> no
> > > >>>>> way
> > > >>>>>  these two changes can be implemented in parallel and
then merged
> > > >>>>> together
> > > >>>>>  easily. So what we will end up with is having to implement
these
> > > >>>> changes
> > > >>>>>  sequentially, fixing all existing tests twice, and essentially
> > > >>>> throwing
> > > >>>>>  away half of the work done because the other part of
the change
> is
> > > >>>>>  re-implemented
> > > >>>>>  - Similar example goes with getting rid of IgniteInternalFuture
> > and
> > > >>>>>  replacing it with CompletableFuture, and any other change
that
> > > >> touches
> > > >>>>> the
> > > >>>>>  asynchronous part of the code.
> > > >>>>>
> > > >>>>> Third, I do not see how this choice affects the UX of
Ignite. The
> > end
> > > >>>> user
> > > >>>>> experience must be fixed in the IEP regardless of the
development
> > > >> process
> > > >>>>> and the fact that we have gaps in this area in Ignite
2.x just
> > > confirms
> > > >>>>> that.
> > > >>>>>
> > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess
if
> > > >>>> reformulate,
> > > >>>>> my point is that we might agree to have a single development
> master
> > > >>>> branch
> > > >>>>> with 'disassembled' end-to-end functionality for some
period of
> > time
> > > to
> > > >>>>> speed up development, and re-assemble the core features
after
> > having
> > > >>>>> submodules tested independently.
> > > >>>>>
> > > >>>>> Nikolay,
> > > >>>>>> We have many features that have to evolve.
> > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
support, etc.
> > > >>>>> This is not very specific. In the end, resources are limited
and
> we
> > > >> will
> > > >>>>> not be able to drive both tracks simultaneously, especially
> after a
> > > >>>> couple
> > > >>>>> of features having been implemented for Ignite 3.0. If
there are
> > > indeed
> > > >>>>> some major changes that we want to do in Ignite 2.x instead
of
> > > putting
> > > >>>>> effort into 3.0 - let's discuss them. I am just not aware
of any,
> > > >> that's
> > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > >>>>>
> > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> breaking
> > > >>>> backward
> > > >>>>> compatibility.
> > > >>>>>> We have many users who are happy with the 2.x with
all it’s
> > issues.
> > > >>>>> These changes will be covered by end-to-end tests and
migrated to
> > > >> Ignite
> > > >>>>> 3.0, so I see no issues here.
> > > >>>>>
> > > >>>>> Finally, Anton & Nikolay
> > > >>>>> I do not have an estimate for this simply because the
activity is
> > > >>>>> community-driven and it depends on the number of people
willing
> to
> > > >>>>> contribute. With the current pace, I would hope to have
an RC of
> > > Ignite
> > > >>>> 3.0
> > > >>>>> to be ready by the end of 2021. My gut feeling is that
by moving
> > with
> > > >>>>> incremental changes, we will not be able to implement
even half
> of
> > > the
> > > >>>>> wishlist by that time.
> > > >>>>> I doubt that releasing several major releases with breaking
> changes
> > > >> will
> > > >>>>> make Ignite users happy either because each upgrade will
cost
> > Ignite
> > > >>>> users
> > > >>>>> money, so the fewer major versions we release, the better.
Thus
> my
> > > wish
> > > >>>> to
> > > >>>>> include all breaking changes in one release.
> > > >>>>>
> > > >>>>> I'll be now quiet for a while, let's see what other community
> > members
> > > >>>>> think.
> > > >>>>>
> > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <
> ptupitsyn@apache.org
> > >:
> > > >>>>>
> > > >>>>>> 1. Rewriting from scratch is never a good idea.
> > > >>>>>> We don't want to follow the path of Netscape and lose
all our
> > users
> > > >>>>>> by the time we have a working 3.0 [1]
> > > >>>>>>
> > > >>>>>> 2. Not sure about new repo - seems like some pain
and no gain,
> > > what's
> > > >>>> the
> > > >>>>>> problem with a branch?
> > > >>>>>>
> > > >>>>>> 3. We should keep existing integration tests when
possible.
> > > >>>>>> We have accumulated a lot of edge case knowledge over
the years,
> > > >>>>>> it is not a good idea to send all of that down the
drain.
> > > >>>>>> Yes, integration tests are slow, but they are the
most valuable.
> > > >>>>>> I think we can move more stuff into nightly runs and
have a fast
> > and
> > > >>>>> modern
> > > >>>>>> basic suite.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Alexey, you are much more familiar with the Ignite
core codebase
> > > than
> > > >>>>> most
> > > >>>>>> of us,
> > > >>>>>> can you please explain in more detail which particular
feature,
> in
> > > >> your
> > > >>>>>> opinion,
> > > >>>>>> mandates this "start from scratch" approach?
> > > >>>>>> Is it really not possible at all to follow a less
radical way?
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > > >>>>>>
> > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> > nizhikov@apache.org
> > > >
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hello, Alexey.
> > > >>>>>>>
> > > >>>>>>> I think that «rewriting from scratch» approach
has a high risk
> to
> > > >>>> make
> > > >>>>>> new
> > > >>>>>>> features unusable.
> > > >>>>>>> At the time Ignite2 was started no-one wants to
do bad UX or
> bad
> > > >>>>>> features.
> > > >>>>>>> Nevertheless, it happen.
> > > >>>>>>>
> > > >>>>>>> I think we can avoid it with the Ignite3 and successors
if we
> > will
> > > >>>> move
> > > >>>>>>> step by step without keeping backward compatibility
> > > >>>>>>> With the step by step approach, we can focus on
each component
> > > >>>>>> separately.
> > > >>>>>>>
> > > >>>>>>>> What new features are we planning to implement
for Ignite 2.x?
> > > >>>>>>>
> > > >>>>>>> We have many features that have to evolve.
> > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper
support, etc.
> > > >>>>>>> We have bugs and issues that can be fixed in 2.x
without
> breaking
> > > >>>>>> backward
> > > >>>>>>> compatibility.
> > > >>>>>>> We have many users who are happy with the 2.x
with all it’s
> > issues.
> > > >>>>>>>
> > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov
<av@apache.org>
> > > >>>>> написал(а):
> > > >>>>>>>>
> > > >>>>>>>> Alexey,
> > > >>>>>>>>
> > > >>>>>>>> Do we have any estimates of how fast we'll
be able to gain
> > > >>>>>>> production-ready
> > > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk
<
> > > >>>>>>> alexey.goncharuk@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Nikolay,
> > > >>>>>>>>>
> > > >>>>>>>>> What new features are we planning to implement
for Ignite
> 2.x?
> > I
> > > >>>>> think
> > > >>>>>>> once
> > > >>>>>>>>> we commence working on Ignite 3.0, we
should gradually cease
> > the
> > > >>>>>>> activity
> > > >>>>>>>>> on Ignite 2.x to mere bugfixes because
such parallel
> > development
> > > >>>>> will
> > > >>>>>> be
> > > >>>>>>>>> overwhelming regardless of how we choose
to proceed.
> > > >>>>>>>>>
> > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay
Izhikov <
> > > nizhikov@apache.org
> > > >>>>> :
> > > >>>>>>>>>
> > > >>>>>>>>>> To be clear:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> I would suggest creating a new
repository for Ignite 3.0
> > > >>>>> (perhaps, a
> > > >>>>>>>>> new
> > > >>>>>>>>>> clean branch, but a new repo looks
nicer to me) and a new
> > Ignite
> > > >>>>> 3.0
> > > >>>>>>>>>> TeamCity project.
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for new Team City project.
> > > >>>>>>>>>> +1 for new branch for Ignite3.
> > > >>>>>>>>>> -1 for new repo.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35,
Nikolay Izhikov <
> > > >>>> NIzhikov.dev@gmail.com
> > > >>>>>>
> > > >>>>>>>>>> написал(а):
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hello, Alexey.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think it will hurt our project
more than help.
> > > >>>>>>>>>>> Developing new features for 2
separate branches with the
> > > >>>> different
> > > >>>>>>> APIs
> > > >>>>>>>>>> and internal structure is overwhelming
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Maybe we should relax a bit requirements
for Ignite3?
> > > >>>>>>>>>>> Maybe we should move step by step
and make Ignite3 with new
> > > >>>>>>>>>> configuration than Ignite4 with new
transactions, etc?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14,
Alexey Goncharuk <
> > > >>>>>>>>> alexey.goncharuk@gmail.com>
> > > >>>>>>>>>> написал(а):
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Igniters,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I wanted to pitch a rather
radical idea regarding the
> Ignite
> > > >>>> 3.0
> > > >>>>>>>>>>>> development which has occurred
to me some time ago.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> We already have several IEPs
targeted to Ignite 3.0 which
> > > imply
> > > >>>>>> major
> > > >>>>>>>>>>>> changes to the codebase (the
change in replication
> protocol
> > > and
> > > >>>>>> thus
> > > >>>>>>>>>>>> transactions, change in binary
format, updated
> metastorage,
> > > >>>> etc).
> > > >>>>>> We
> > > >>>>>>>>>> also
> > > >>>>>>>>>>>> planned significant changes
in public APIs: configuration
> > > >>>> format
> > > >>>>>>>>> change,
> > > >>>>>>>>>>>> improvements in cache APIs,
SQL APIs, transaction mode
> > rework.
> > > >>>>> The
> > > >>>>>>>>>> wishlist
> > > >>>>>>>>>>>> of changes for Ignite 3.0
is huge.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> So, I was wondering whether
it makes sense to try to
> change
> > > the
> > > >>>>> old
> > > >>>>>>>>>>>> codebase, or start with a
new baseline and move old pieces
> > of
> > > >>>>> code
> > > >>>>>>>>> that
> > > >>>>>>>>>> do
> > > >>>>>>>>>>>> not require significant rework.
Personally, I would go
> with
> > > the
> > > >>>>>>> second
> > > >>>>>>>>>>>> option for the following reasons:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - We have a chance to shift
the development paradigm in
> the
> > > >>>>> project
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> introduce the practice of
true unit-tests. In the new
> > baseline
> > > >>>> at
> > > >>>>>> the
> > > >>>>>>>>>>>> beginning there will be no
ability to run an end-to-end
> > > >>>> scenario,
> > > >>>>>>>>> thus
> > > >>>>>>>>>> we
> > > >>>>>>>>>>>> will be forced to write unit-tests.
So far, such practice
> > was
> > > >>>>> hard
> > > >>>>>> to
> > > >>>>>>>>>>>> implement because of tight
coupling between Ignite
> > components
> > > >>>> and
> > > >>>>>>>>>> inability
> > > >>>>>>>>>>>> to instantiate components
without an instance of
> > > KernalContext.
> > > >>>>> For
> > > >>>>>>>>>>>> example, we should be able
to thoroughly test internal
> > > >>>>> primitives,
> > > >>>>>>>>>> such as
> > > >>>>>>>>>>>> replication protocol (without
actual communication),
> > > >>>> distributed
> > > >>>>>>>>>>>> metastorage contracts, persistence
layer, etc.
> > > >>>>>>>>>>>> - We will significantly reduce
the development cycle in
> the
> > > >>>>>> beginning
> > > >>>>>>>>>>>> (right now the RunAll takes
two hours of astronomical time
> > > with
> > > >>>>>> empty
> > > >>>>>>>>>> TC;
> > > >>>>>>>>>>>> in the new approach developer
will be able to run ALL
> tests
> > > >>>>> locally
> > > >>>>>>>>> in
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>> matter of minutes)
> > > >>>>>>>>>>>> - We can get rid of TC bot
and enforce green TC by
> > integrating
> > > >>>> TC
> > > >>>>>>>>> build
> > > >>>>>>>>>>>> results with GitHub PRs (the
same way Travis is currently
> > > >>>>>> integrated
> > > >>>>>>>>>> to PR
> > > >>>>>>>>>>>> check). We should restrict
PR merge without a TC check
> > > >>>>>>>>>>>> - We will still have to re-write
all tests, but only once.
> > If
> > > >>>> we
> > > >>>>>> try
> > > >>>>>>>>> to
> > > >>>>>>>>>>>> modify the old codebase, we
would need to modify all the
> > tests
> > > >>>>> for
> > > >>>>>>>>>> every
> > > >>>>>>>>>>>> major change (public API change,
configuration change)
> > > >>>>>>>>>>>> - We will have fewer conflicts
when working together. For
> > > >>>>> example,
> > > >>>>>> I
> > > >>>>>>>>>>>> cannot imagine how one would
merge two changes of getting
> > rid
> > > >>>> of
> > > >>>>>>>>>>>> IgniteFuture and changes in
replication protocol, for
> > example
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Technically, I would suggest
creating a new repository for
> > > >>>> Ignite
> > > >>>>>> 3.0
> > > >>>>>>>>>>>> (perhaps, a new clean branch,
but a new repo looks nicer
> to
> > > me)
> > > >>>>>> and a
> > > >>>>>>>>>> new
> > > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> While it may seem quite radical,
I do believe that this
> > > >>>> approach
> > > >>>>>> will
> > > >>>>>>>>>> give
> > > >>>>>>>>>>>> us more benefits than trying
to make such major changes in
> > the
> > > >>>>>>>>> existing
> > > >>>>>>>>>>>> codebase. If needed, let's
schedule a discord chat like
> > before
> > > >>>> to
> > > >>>>>>>>>> discuss
> > > >>>>>>>>>>>> this.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Best regards,
> > > >>>> Andrey V. Mashenkov
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message