ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikita Ivanov <nivano...@gmail.com>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Tue, 03 Nov 2020 07:12:49 GMT
+1 on having a separate repo. Make the work cleaner and more effective.

--
Nikita Ivanov



On Mon, Nov 2, 2020 at 11:09 PM Nikolay Izhikov <nizhikov@apache.org> wrote:

> Igniters, should we have a call for this topic?
>
> > 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org>
> написал(а):
> >
> >> not intend to rewrite everything from scratch
> >
> >> Every single test from Ignite 2.x should be moved to Ignite 3
> >> regardless of how we choose to proceed.
> >
> > Alexey, thank you for the explanation, this addresses all of my concerns.
> >
> >
> >
> >
> >
> > On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> andrey.mashenkov@gmail.com>
> > wrote:
> >
> >> Hi, Igniters.
> >>
> >> * AFAIU, we need a new repo if we want to apply different restrictions
> to
> >> pull requests,
> >> otherwise I see no difference for myself.
> >> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >> checks mandatory.
> >>
> >> I think that relaxed requirements here will lead to bad product quality.
> >>
> >> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >> During active development tests will be broken most of time, so,
> >> I'd port them e.g. suite-by-suite once we will have a stable and
> featured
> >> environment to run them and of course make test's code clear and avoid
> >> bad/non-relevant ones.
> >>
> >> * I like bottom-up approach.
> >> With it we could make a better framework. I mean clear component
> lifecycle,
> >> component wiring mechanics, general methods to approach core components
> >> such as exchange/communication
> >> to avoid code mess like we have in ExchangeFuture with all these custom
> >> callbacks for each component, interfaces like
> >> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >> a pack of
> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >> and so on in various unexpected places.
> >> Hope, we will be able to port most of the good code to the new framework
> >> version.
> >>
> >>
> >>
> >> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >> alexey.goncharuk@gmail.com>
> >> wrote:
> >>
> >>> Nikolay, Pavel,
> >>>
> >>> Thanks for the feedback! First of all, I wanted to stress that I do not
> >>> intend to rewrite everything from scratch (I never used this phrase).
> >> There
> >>> are significant parts of code that would be moved with minimal
> >>> modifications. Second, I never said that we will get rid of the old
> tests
> >>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> >>> regardless of how we choose to proceed.
> >>>
> >>> My point is that for some parts of the code a clean bottom-up
> >>> implementation will be cheaper in many ways. Let me give you a few
> >> concrete
> >>> examples:
> >>>
> >>>   - I think no one can object that we need a cleanly separated
> >> persistence
> >>>   layer for Ignite. There is a very raw draft IEP for this already. On
> >> the
> >>>   other hand, I think we also can agree that we need a split-brain
> >>> resistant
> >>>   replication protocol for caches. There is also an IEP for this.
> >> Neither
> >>> of
> >>>   the changes is a good fit for 2.x because they are likely to
> introduce
> >>>   breaking changes in the persistence layer, configuration and
> behavior.
> >>>   Additionally, these components are now tightly coupled, so there is
> no
> >>> way
> >>>   these two changes can be implemented in parallel and then merged
> >>> together
> >>>   easily. So what we will end up with is having to implement these
> >> changes
> >>>   sequentially, fixing all existing tests twice, and essentially
> >> throwing
> >>>   away half of the work done because the other part of the change is
> >>>   re-implemented
> >>>   - Similar example goes with getting rid of IgniteInternalFuture and
> >>>   replacing it with CompletableFuture, and any other change that
> touches
> >>> the
> >>>   asynchronous part of the code.
> >>>
> >>> Third, I do not see how this choice affects the UX of Ignite. The end
> >> user
> >>> experience must be fixed in the IEP regardless of the development
> process
> >>> and the fact that we have gaps in this area in Ignite 2.x just confirms
> >>> that.
> >>>
> >>> Pavel, agree that a repo/branch is a technicality, I guess if
> >> reformulate,
> >>> my point is that we might agree to have a single development master
> >> branch
> >>> with 'disassembled' end-to-end functionality for some period of time to
> >>> speed up development, and re-assemble the core features after having
> >>> submodules tested independently.
> >>>
> >>> Nikolay,
> >>>> We have many features that have to evolve.
> >>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>> This is not very specific. In the end, resources are limited and we
> will
> >>> not be able to drive both tracks simultaneously, especially after a
> >> couple
> >>> of features having been implemented for Ignite 3.0. If there are indeed
> >>> some major changes that we want to do in Ignite 2.x instead of putting
> >>> effort into 3.0 - let's discuss them. I am just not aware of any,
> that's
> >>> why I am eager to move forward with Ignite 3.0.
> >>>
> >>>> We have bugs and issues that can be fixed in 2.x without breaking
> >> backward
> >>> compatibility.
> >>>> We have many users who are happy with the 2.x with all it’s issues.
> >>> These changes will be covered by end-to-end tests and migrated to
> Ignite
> >>> 3.0, so I see no issues here.
> >>>
> >>> Finally, Anton & Nikolay
> >>> I do not have an estimate for this simply because the activity is
> >>> community-driven and it depends on the number of people willing to
> >>> contribute. With the current pace, I would hope to have an RC of Ignite
> >> 3.0
> >>> to be ready by the end of 2021. My gut feeling is that by moving with
> >>> incremental changes, we will not be able to implement even half of the
> >>> wishlist by that time.
> >>> I doubt that releasing several major releases with breaking changes
> will
> >>> make Ignite users happy either because each upgrade will cost Ignite
> >> users
> >>> money, so the fewer major versions we release, the better. Thus my wish
> >> to
> >>> include all breaking changes in one release.
> >>>
> >>> I'll be now quiet for a while, let's see what other community members
> >>> think.
> >>>
> >>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org>:
> >>>
> >>>> 1. Rewriting from scratch is never a good idea.
> >>>> We don't want to follow the path of Netscape and lose all our users
> >>>> by the time we have a working 3.0 [1]
> >>>>
> >>>> 2. Not sure about new repo - seems like some pain and no gain, what's
> >> the
> >>>> problem with a branch?
> >>>>
> >>>> 3. We should keep existing integration tests when possible.
> >>>> We have accumulated a lot of edge case knowledge over the years,
> >>>> it is not a good idea to send all of that down the drain.
> >>>> Yes, integration tests are slow, but they are the most valuable.
> >>>> I think we can move more stuff into nightly runs and have a fast and
> >>> modern
> >>>> basic suite.
> >>>>
> >>>>
> >>>> Alexey, you are much more familiar with the Ignite core codebase than
> >>> most
> >>>> of us,
> >>>> can you please explain in more detail which particular feature, in
> your
> >>>> opinion,
> >>>> mandates this "start from scratch" approach?
> >>>> Is it really not possible at all to follow a less radical way?
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>
> >>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hello, Alexey.
> >>>>>
> >>>>> I think that «rewriting from scratch» approach has a high risk
to
> >> make
> >>>> new
> >>>>> features unusable.
> >>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
> >>>> features.
> >>>>> Nevertheless, it happen.
> >>>>>
> >>>>> I think we can avoid it with the Ignite3 and successors if we will
> >> move
> >>>>> step by step without keeping backward compatibility
> >>>>> With the step by step approach, we can focus on each component
> >>>> separately.
> >>>>>
> >>>>>> What new features are we planning to implement for Ignite 2.x?
> >>>>>
> >>>>> We have many features that have to evolve.
> >>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>> We have many users who are happy with the 2.x with all it’s issues.
> >>>>>
> >>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org>
> >>> написал(а):
> >>>>>>
> >>>>>> Alexey,
> >>>>>>
> >>>>>> Do we have any estimates of how fast we'll be able to gain
> >>>>> production-ready
> >>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>> alexey.goncharuk@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Nikolay,
> >>>>>>>
> >>>>>>> What new features are we planning to implement for Ignite
2.x? I
> >>> think
> >>>>> once
> >>>>>>> we commence working on Ignite 3.0, we should gradually cease
the
> >>>>> activity
> >>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
> >>> will
> >>>> be
> >>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>
> >>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
> >>> :
> >>>>>>>
> >>>>>>>> To be clear:
> >>>>>>>>
> >>>>>>>>> I would suggest creating a new repository for Ignite
3.0
> >>> (perhaps, a
> >>>>>>> new
> >>>>>>>> clean branch, but a new repo looks nicer to me) and
a new Ignite
> >>> 3.0
> >>>>>>>> TeamCity project.
> >>>>>>>>
> >>>>>>>> +1 for new Team City project.
> >>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>> -1 for new repo.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov
<
> >> NIzhikov.dev@gmail.com
> >>>>
> >>>>>>>> написал(а):
> >>>>>>>>>
> >>>>>>>>> Hello, Alexey.
> >>>>>>>>>
> >>>>>>>>> I think it will hurt our project more than help.
> >>>>>>>>> Developing new features for 2 separate branches
with the
> >> different
> >>>>> APIs
> >>>>>>>> and internal structure is overwhelming
> >>>>>>>>>
> >>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
> >>>>>>>>> Maybe we should move step by step and make Ignite3
with new
> >>>>>>>> configuration than Ignite4 with new transactions, etc?
> >>>>>>>>>
> >>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk
<
> >>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>> написал(а):
> >>>>>>>>>>
> >>>>>>>>>> Igniters,
> >>>>>>>>>>
> >>>>>>>>>> I wanted to pitch a rather radical idea regarding
the Ignite
> >> 3.0
> >>>>>>>>>> development which has occurred to me some time
ago.
> >>>>>>>>>>
> >>>>>>>>>> We already have several IEPs targeted to Ignite
3.0 which imply
> >>>> major
> >>>>>>>>>> changes to the codebase (the change in replication
protocol and
> >>>> thus
> >>>>>>>>>> transactions, change in binary format, updated
metastorage,
> >> etc).
> >>>> We
> >>>>>>>> also
> >>>>>>>>>> planned significant changes in public APIs:
configuration
> >> format
> >>>>>>> change,
> >>>>>>>>>> improvements in cache APIs, SQL APIs, transaction
mode rework.
> >>> The
> >>>>>>>> wishlist
> >>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>
> >>>>>>>>>> So, I was wondering whether it makes sense to
try to change the
> >>> old
> >>>>>>>>>> codebase, or start with a new baseline and move
old pieces of
> >>> code
> >>>>>>> that
> >>>>>>>> do
> >>>>>>>>>> not require significant rework. Personally,
I would go with the
> >>>>> second
> >>>>>>>>>> option for the following reasons:
> >>>>>>>>>>
> >>>>>>>>>> - We have a chance to shift the development
paradigm in the
> >>> project
> >>>>>>> and
> >>>>>>>>>> introduce the practice of true unit-tests. In
the new baseline
> >> at
> >>>> the
> >>>>>>>>>> beginning there will be no ability to run an
end-to-end
> >> scenario,
> >>>>>>> thus
> >>>>>>>> we
> >>>>>>>>>> will be forced to write unit-tests. So far,
such practice was
> >>> hard
> >>>> to
> >>>>>>>>>> implement because of tight coupling between
Ignite components
> >> and
> >>>>>>>> inability
> >>>>>>>>>> to instantiate components without an instance
of KernalContext.
> >>> For
> >>>>>>>>>> example, we should be able to thoroughly test
internal
> >>> primitives,
> >>>>>>>> such as
> >>>>>>>>>> replication protocol (without actual communication),
> >> distributed
> >>>>>>>>>> metastorage contracts, persistence layer, etc.
> >>>>>>>>>> - We will significantly reduce the development
cycle in the
> >>>> beginning
> >>>>>>>>>> (right now the RunAll takes two hours of astronomical
time with
> >>>> empty
> >>>>>>>> TC;
> >>>>>>>>>> in the new approach developer will be able to
run ALL tests
> >>> locally
> >>>>>>> in
> >>>>>>>> a
> >>>>>>>>>> matter of minutes)
> >>>>>>>>>> - We can get rid of TC bot and enforce green
TC by integrating
> >> TC
> >>>>>>> build
> >>>>>>>>>> results with GitHub PRs (the same way Travis
is currently
> >>>> integrated
> >>>>>>>> to PR
> >>>>>>>>>> check). We should restrict PR merge without
a TC check
> >>>>>>>>>> - We will still have to re-write all tests,
but only once. If
> >> we
> >>>> try
> >>>>>>> to
> >>>>>>>>>> modify the old codebase, we would need to modify
all the tests
> >>> for
> >>>>>>>> every
> >>>>>>>>>> major change (public API change, configuration
change)
> >>>>>>>>>> - We will have fewer conflicts when working
together. For
> >>> example,
> >>>> I
> >>>>>>>>>> cannot imagine how one would merge two changes
of getting rid
> >> of
> >>>>>>>>>> IgniteFuture and changes in replication protocol,
for example
> >>>>>>>>>>
> >>>>>>>>>> Technically, I would suggest creating a new
repository for
> >> Ignite
> >>>> 3.0
> >>>>>>>>>> (perhaps, a new clean branch, but a new repo
looks nicer to me)
> >>>> and a
> >>>>>>>> new
> >>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> While it may seem quite radical, I do believe
that this
> >> approach
> >>>> will
> >>>>>>>> give
> >>>>>>>>>> us more benefits than trying to make such major
changes in the
> >>>>>>> existing
> >>>>>>>>>> codebase. If needed, let's schedule a discord
chat like before
> >> to
> >>>>>>>> discuss
> >>>>>>>>>> this.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrey V. Mashenkov
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message