ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Tupitsyn <ptupit...@apache.org>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Mon, 02 Nov 2020 15:53:13 GMT
> not intend to rewrite everything from scratch

> Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.

Alexey, thank you for the explanation, this addresses all of my concerns.





On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <andrey.mashenkov@gmail.com>
wrote:

> Hi, Igniters.
>
> * AFAIU, we need a new repo if we want to apply different restrictions to
> pull requests,
> otherwise I see no difference for myself.
> E.g. make static analysis (do we have?), compile, styles, and javadoc
> checks mandatory.
>
> I think that relaxed requirements here will lead to bad product quality.
>
> * Agree with Pavel, we should 'keep' integrations tests somehow.
> During active development tests will be broken most of time, so,
> I'd port them e.g. suite-by-suite once we will have a stable and featured
> environment to run them and of course make test's code clear and avoid
> bad/non-relevant ones.
>
> * I like bottom-up approach.
> With it we could make a better framework. I mean clear component lifecycle,
> component wiring mechanics, general methods to approach core components
> such as exchange/communication
> to avoid code mess like we have in ExchangeFuture with all these custom
> callbacks for each component, interfaces like
> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> and so on in various unexpected places.
> Hope, we will be able to port most of the good code to the new framework
> version.
>
>
>
> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> alexey.goncharuk@gmail.com>
> wrote:
>
> > Nikolay, Pavel,
> >
> > Thanks for the feedback! First of all, I wanted to stress that I do not
> > intend to rewrite everything from scratch (I never used this phrase).
> There
> > are significant parts of code that would be moved with minimal
> > modifications. Second, I never said that we will get rid of the old tests
> > codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> > regardless of how we choose to proceed.
> >
> > My point is that for some parts of the code a clean bottom-up
> > implementation will be cheaper in many ways. Let me give you a few
> concrete
> > examples:
> >
> >    - I think no one can object that we need a cleanly separated
> persistence
> >    layer for Ignite. There is a very raw draft IEP for this already. On
> the
> >    other hand, I think we also can agree that we need a split-brain
> > resistant
> >    replication protocol for caches. There is also an IEP for this.
> Neither
> > of
> >    the changes is a good fit for 2.x because they are likely to introduce
> >    breaking changes in the persistence layer, configuration and behavior.
> >    Additionally, these components are now tightly coupled, so there is no
> > way
> >    these two changes can be implemented in parallel and then merged
> > together
> >    easily. So what we will end up with is having to implement these
> changes
> >    sequentially, fixing all existing tests twice, and essentially
> throwing
> >    away half of the work done because the other part of the change is
> >    re-implemented
> >    - Similar example goes with getting rid of IgniteInternalFuture and
> >    replacing it with CompletableFuture, and any other change that touches
> > the
> >    asynchronous part of the code.
> >
> > Third, I do not see how this choice affects the UX of Ignite. The end
> user
> > experience must be fixed in the IEP regardless of the development process
> > and the fact that we have gaps in this area in Ignite 2.x just confirms
> > that.
> >
> > Pavel, agree that a repo/branch is a technicality, I guess if
> reformulate,
> > my point is that we might agree to have a single development master
> branch
> > with 'disassembled' end-to-end functionality for some period of time to
> > speed up development, and re-assemble the core features after having
> > submodules tested independently.
> >
> > Nikolay,
> >   >We have many features that have to evolve.
> >   >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > This is not very specific. In the end, resources are limited and we will
> > not be able to drive both tracks simultaneously, especially after a
> couple
> > of features having been implemented for Ignite 3.0. If there are indeed
> > some major changes that we want to do in Ignite 2.x instead of putting
> > effort into 3.0 - let's discuss them. I am just not aware of any, that's
> > why I am eager to move forward with Ignite 3.0.
> >
> > >We have bugs and issues that can be fixed in 2.x without breaking
> backward
> > compatibility.
> > >We have many users who are happy with the 2.x with all it’s issues.
> > These changes will be covered by end-to-end tests and migrated to Ignite
> > 3.0, so I see no issues here.
> >
> > Finally, Anton & Nikolay
> > I do not have an estimate for this simply because the activity is
> > community-driven and it depends on the number of people willing to
> > contribute. With the current pace, I would hope to have an RC of Ignite
> 3.0
> > to be ready by the end of 2021. My gut feeling is that by moving with
> > incremental changes, we will not be able to implement even half of the
> > wishlist by that time.
> > I doubt that releasing several major releases with breaking changes will
> > make Ignite users happy either because each upgrade will cost Ignite
> users
> > money, so the fewer major versions we release, the better. Thus my wish
> to
> > include all breaking changes in one release.
> >
> > I'll be now quiet for a while, let's see what other community members
> > think.
> >
> > пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org>:
> >
> > > 1. Rewriting from scratch is never a good idea.
> > > We don't want to follow the path of Netscape and lose all our users
> > > by the time we have a working 3.0 [1]
> > >
> > > 2. Not sure about new repo - seems like some pain and no gain, what's
> the
> > > problem with a branch?
> > >
> > > 3. We should keep existing integration tests when possible.
> > > We have accumulated a lot of edge case knowledge over the years,
> > > it is not a good idea to send all of that down the drain.
> > > Yes, integration tests are slow, but they are the most valuable.
> > > I think we can move more stuff into nightly runs and have a fast and
> > modern
> > > basic suite.
> > >
> > >
> > > Alexey, you are much more familiar with the Ignite core codebase than
> > most
> > > of us,
> > > can you please explain in more detail which particular feature, in your
> > > opinion,
> > > mandates this "start from scratch" approach?
> > > Is it really not possible at all to follow a less radical way?
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > >
> > > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org>
> > > wrote:
> > >
> > > > Hello, Alexey.
> > > >
> > > > I think that «rewriting from scratch» approach has a high risk to
> make
> > > new
> > > > features unusable.
> > > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > > features.
> > > > Nevertheless, it happen.
> > > >
> > > > I think we can avoid it with the Ignite3 and successors if we will
> move
> > > > step by step without keeping backward compatibility
> > > > With the step by step approach, we can focus on each component
> > > separately.
> > > >
> > > > > What new features are we planning to implement for Ignite 2.x?
> > > >
> > > > We have many features that have to evolve.
> > > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > > We have bugs and issues that can be fixed in 2.x without breaking
> > > backward
> > > > compatibility.
> > > > We have many users who are happy with the 2.x with all it’s issues.
> > > >
> > > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org>
> > написал(а):
> > > > >
> > > > > Alexey,
> > > > >
> > > > > Do we have any estimates of how fast we'll be able to gain
> > > > production-ready
> > > > > AI 3.0 in case of a "new repo" choice?
> > > > >
> > > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > > alexey.goncharuk@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Nikolay,
> > > > >>
> > > > >> What new features are we planning to implement for Ignite 2.x?
I
> > think
> > > > once
> > > > >> we commence working on Ignite 3.0, we should gradually cease
the
> > > > activity
> > > > >> on Ignite 2.x to mere bugfixes because such parallel development
> > will
> > > be
> > > > >> overwhelming regardless of how we choose to proceed.
> > > > >>
> > > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
> >:
> > > > >>
> > > > >>> To be clear:
> > > > >>>
> > > > >>>> I would suggest creating a new repository for Ignite
3.0
> > (perhaps, a
> > > > >> new
> > > > >>> clean branch, but a new repo looks nicer to me) and a new
Ignite
> > 3.0
> > > > >>> TeamCity project.
> > > > >>>
> > > > >>> +1 for new Team City project.
> > > > >>> +1 for new branch for Ignite3.
> > > > >>> -1 for new repo.
> > > > >>>
> > > > >>>
> > > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
> NIzhikov.dev@gmail.com
> > >
> > > > >>> написал(а):
> > > > >>>>
> > > > >>>> Hello, Alexey.
> > > > >>>>
> > > > >>>> I think it will hurt our project more than help.
> > > > >>>> Developing new features for 2 separate branches with
the
> different
> > > > APIs
> > > > >>> and internal structure is overwhelming
> > > > >>>>
> > > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > > >>>> Maybe we should move step by step and make Ignite3 with
new
> > > > >>> configuration than Ignite4 with new transactions, etc?
> > > > >>>>
> > > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk
<
> > > > >> alexey.goncharuk@gmail.com>
> > > > >>> написал(а):
> > > > >>>>>
> > > > >>>>> Igniters,
> > > > >>>>>
> > > > >>>>> I wanted to pitch a rather radical idea regarding
the Ignite
> 3.0
> > > > >>>>> development which has occurred to me some time ago.
> > > > >>>>>
> > > > >>>>> We already have several IEPs targeted to Ignite 3.0
which imply
> > > major
> > > > >>>>> changes to the codebase (the change in replication
protocol and
> > > thus
> > > > >>>>> transactions, change in binary format, updated metastorage,
> etc).
> > > We
> > > > >>> also
> > > > >>>>> planned significant changes in public APIs: configuration
> format
> > > > >> change,
> > > > >>>>> improvements in cache APIs, SQL APIs, transaction
mode rework.
> > The
> > > > >>> wishlist
> > > > >>>>> of changes for Ignite 3.0 is huge.
> > > > >>>>>
> > > > >>>>> So, I was wondering whether it makes sense to try
to change the
> > old
> > > > >>>>> codebase, or start with a new baseline and move old
pieces of
> > code
> > > > >> that
> > > > >>> do
> > > > >>>>> not require significant rework. Personally, I would
go with the
> > > > second
> > > > >>>>> option for the following reasons:
> > > > >>>>>
> > > > >>>>> - We have a chance to shift the development paradigm
in the
> > project
> > > > >> and
> > > > >>>>> introduce the practice of true unit-tests. In the
new baseline
> at
> > > the
> > > > >>>>> beginning there will be no ability to run an end-to-end
> scenario,
> > > > >> thus
> > > > >>> we
> > > > >>>>> will be forced to write unit-tests. So far, such
practice was
> > hard
> > > to
> > > > >>>>> implement because of tight coupling between Ignite
components
> and
> > > > >>> inability
> > > > >>>>> to instantiate components without an instance of
KernalContext.
> > For
> > > > >>>>> example, we should be able to thoroughly test internal
> > primitives,
> > > > >>> such as
> > > > >>>>> replication protocol (without actual communication),
> distributed
> > > > >>>>> metastorage contracts, persistence layer, etc.
> > > > >>>>> - We will significantly reduce the development cycle
in the
> > > beginning
> > > > >>>>> (right now the RunAll takes two hours of astronomical
time with
> > > empty
> > > > >>> TC;
> > > > >>>>> in the new approach developer will be able to run
ALL tests
> > locally
> > > > >> in
> > > > >>> a
> > > > >>>>> matter of minutes)
> > > > >>>>> - We can get rid of TC bot and enforce green TC by
integrating
> TC
> > > > >> build
> > > > >>>>> results with GitHub PRs (the same way Travis is currently
> > > integrated
> > > > >>> to PR
> > > > >>>>> check). We should restrict PR merge without a TC
check
> > > > >>>>> - We will still have to re-write all tests, but only
once. If
> we
> > > try
> > > > >> to
> > > > >>>>> modify the old codebase, we would need to modify
all the tests
> > for
> > > > >>> every
> > > > >>>>> major change (public API change, configuration change)
> > > > >>>>> - We will have fewer conflicts when working together.
For
> > example,
> > > I
> > > > >>>>> cannot imagine how one would merge two changes of
getting rid
> of
> > > > >>>>> IgniteFuture and changes in replication protocol,
for example
> > > > >>>>>
> > > > >>>>> Technically, I would suggest creating a new repository
for
> Ignite
> > > 3.0
> > > > >>>>> (perhaps, a new clean branch, but a new repo looks
nicer to me)
> > > and a
> > > > >>> new
> > > > >>>>> Ignite 3.0 TeamCity project.
> > > > >>>>>
> > > > >>>>> While it may seem quite radical, I do believe that
this
> approach
> > > will
> > > > >>> give
> > > > >>>>> us more benefits than trying to make such major changes
in the
> > > > >> existing
> > > > >>>>> codebase. If needed, let's schedule a discord chat
like before
> to
> > > > >>> discuss
> > > > >>>>> this.
> > > > >>>>>
> > > > >>>>> WDYT?
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message