ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Mon, 02 Nov 2020 15:42:46 GMT
Hi, Igniters.

* AFAIU, we need a new repo if we want to apply different restrictions to
pull requests,
otherwise I see no difference for myself.
E.g. make static analysis (do we have?), compile, styles, and javadoc
checks mandatory.

I think that relaxed requirements here will lead to bad product quality.

* Agree with Pavel, we should 'keep' integrations tests somehow.
During active development tests will be broken most of time, so,
I'd port them e.g. suite-by-suite once we will have a stable and featured
environment to run them and of course make test's code clear and avoid
bad/non-relevant ones.

* I like bottom-up approach.
With it we could make a better framework. I mean clear component lifecycle,
component wiring mechanics, general methods to approach core components
such as exchange/communication
to avoid code mess like we have in ExchangeFuture with all these custom
callbacks for each component, interfaces like
PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
and so on in various unexpected places.
Hope, we will be able to port most of the good code to the new framework
version.



On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <alexey.goncharuk@gmail.com>
wrote:

> Nikolay, Pavel,
>
> Thanks for the feedback! First of all, I wanted to stress that I do not
> intend to rewrite everything from scratch (I never used this phrase). There
> are significant parts of code that would be moved with minimal
> modifications. Second, I never said that we will get rid of the old tests
> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
> regardless of how we choose to proceed.
>
> My point is that for some parts of the code a clean bottom-up
> implementation will be cheaper in many ways. Let me give you a few concrete
> examples:
>
>    - I think no one can object that we need a cleanly separated persistence
>    layer for Ignite. There is a very raw draft IEP for this already. On the
>    other hand, I think we also can agree that we need a split-brain
> resistant
>    replication protocol for caches. There is also an IEP for this. Neither
> of
>    the changes is a good fit for 2.x because they are likely to introduce
>    breaking changes in the persistence layer, configuration and behavior.
>    Additionally, these components are now tightly coupled, so there is no
> way
>    these two changes can be implemented in parallel and then merged
> together
>    easily. So what we will end up with is having to implement these changes
>    sequentially, fixing all existing tests twice, and essentially throwing
>    away half of the work done because the other part of the change is
>    re-implemented
>    - Similar example goes with getting rid of IgniteInternalFuture and
>    replacing it with CompletableFuture, and any other change that touches
> the
>    asynchronous part of the code.
>
> Third, I do not see how this choice affects the UX of Ignite. The end user
> experience must be fixed in the IEP regardless of the development process
> and the fact that we have gaps in this area in Ignite 2.x just confirms
> that.
>
> Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
> my point is that we might agree to have a single development master branch
> with 'disassembled' end-to-end functionality for some period of time to
> speed up development, and re-assemble the core features after having
> submodules tested independently.
>
> Nikolay,
>   >We have many features that have to evolve.
>   >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> This is not very specific. In the end, resources are limited and we will
> not be able to drive both tracks simultaneously, especially after a couple
> of features having been implemented for Ignite 3.0. If there are indeed
> some major changes that we want to do in Ignite 2.x instead of putting
> effort into 3.0 - let's discuss them. I am just not aware of any, that's
> why I am eager to move forward with Ignite 3.0.
>
> >We have bugs and issues that can be fixed in 2.x without breaking backward
> compatibility.
> >We have many users who are happy with the 2.x with all it’s issues.
> These changes will be covered by end-to-end tests and migrated to Ignite
> 3.0, so I see no issues here.
>
> Finally, Anton & Nikolay
> I do not have an estimate for this simply because the activity is
> community-driven and it depends on the number of people willing to
> contribute. With the current pace, I would hope to have an RC of Ignite 3.0
> to be ready by the end of 2021. My gut feeling is that by moving with
> incremental changes, we will not be able to implement even half of the
> wishlist by that time.
> I doubt that releasing several major releases with breaking changes will
> make Ignite users happy either because each upgrade will cost Ignite users
> money, so the fewer major versions we release, the better. Thus my wish to
> include all breaking changes in one release.
>
> I'll be now quiet for a while, let's see what other community members
> think.
>
> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org>:
>
> > 1. Rewriting from scratch is never a good idea.
> > We don't want to follow the path of Netscape and lose all our users
> > by the time we have a working 3.0 [1]
> >
> > 2. Not sure about new repo - seems like some pain and no gain, what's the
> > problem with a branch?
> >
> > 3. We should keep existing integration tests when possible.
> > We have accumulated a lot of edge case knowledge over the years,
> > it is not a good idea to send all of that down the drain.
> > Yes, integration tests are slow, but they are the most valuable.
> > I think we can move more stuff into nightly runs and have a fast and
> modern
> > basic suite.
> >
> >
> > Alexey, you are much more familiar with the Ignite core codebase than
> most
> > of us,
> > can you please explain in more detail which particular feature, in your
> > opinion,
> > mandates this "start from scratch" approach?
> > Is it really not possible at all to follow a less radical way?
> >
> >
> > [1]
> >
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >
> > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org>
> > wrote:
> >
> > > Hello, Alexey.
> > >
> > > I think that «rewriting from scratch» approach has a high risk to make
> > new
> > > features unusable.
> > > At the time Ignite2 was started no-one wants to do bad UX or bad
> > features.
> > > Nevertheless, it happen.
> > >
> > > I think we can avoid it with the Ignite3 and successors if we will move
> > > step by step without keeping backward compatibility
> > > With the step by step approach, we can focus on each component
> > separately.
> > >
> > > > What new features are we planning to implement for Ignite 2.x?
> > >
> > > We have many features that have to evolve.
> > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > > We have bugs and issues that can be fixed in 2.x without breaking
> > backward
> > > compatibility.
> > > We have many users who are happy with the 2.x with all it’s issues.
> > >
> > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org>
> написал(а):
> > > >
> > > > Alexey,
> > > >
> > > > Do we have any estimates of how fast we'll be able to gain
> > > production-ready
> > > > AI 3.0 in case of a "new repo" choice?
> > > >
> > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com>
> > > > wrote:
> > > >
> > > >> Nikolay,
> > > >>
> > > >> What new features are we planning to implement for Ignite 2.x? I
> think
> > > once
> > > >> we commence working on Ignite 3.0, we should gradually cease the
> > > activity
> > > >> on Ignite 2.x to mere bugfixes because such parallel development
> will
> > be
> > > >> overwhelming regardless of how we choose to proceed.
> > > >>
> > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org>:
> > > >>
> > > >>> To be clear:
> > > >>>
> > > >>>> I would suggest creating a new repository for Ignite 3.0
> (perhaps, a
> > > >> new
> > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite
> 3.0
> > > >>> TeamCity project.
> > > >>>
> > > >>> +1 for new Team City project.
> > > >>> +1 for new branch for Ignite3.
> > > >>> -1 for new repo.
> > > >>>
> > > >>>
> > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NIzhikov.dev@gmail.com
> >
> > > >>> написал(а):
> > > >>>>
> > > >>>> Hello, Alexey.
> > > >>>>
> > > >>>> I think it will hurt our project more than help.
> > > >>>> Developing new features for 2 separate branches with the different
> > > APIs
> > > >>> and internal structure is overwhelming
> > > >>>>
> > > >>>> Maybe we should relax a bit requirements for Ignite3?
> > > >>>> Maybe we should move step by step and make Ignite3 with new
> > > >>> configuration than Ignite4 with new transactions, etc?
> > > >>>>
> > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > > >> alexey.goncharuk@gmail.com>
> > > >>> написал(а):
> > > >>>>>
> > > >>>>> Igniters,
> > > >>>>>
> > > >>>>> I wanted to pitch a rather radical idea regarding the
Ignite 3.0
> > > >>>>> development which has occurred to me some time ago.
> > > >>>>>
> > > >>>>> We already have several IEPs targeted to Ignite 3.0 which
imply
> > major
> > > >>>>> changes to the codebase (the change in replication protocol
and
> > thus
> > > >>>>> transactions, change in binary format, updated metastorage,
etc).
> > We
> > > >>> also
> > > >>>>> planned significant changes in public APIs: configuration
format
> > > >> change,
> > > >>>>> improvements in cache APIs, SQL APIs, transaction mode
rework.
> The
> > > >>> wishlist
> > > >>>>> of changes for Ignite 3.0 is huge.
> > > >>>>>
> > > >>>>> So, I was wondering whether it makes sense to try to change
the
> old
> > > >>>>> codebase, or start with a new baseline and move old pieces
of
> code
> > > >> that
> > > >>> do
> > > >>>>> not require significant rework. Personally, I would go
with the
> > > second
> > > >>>>> option for the following reasons:
> > > >>>>>
> > > >>>>> - We have a chance to shift the development paradigm in
the
> project
> > > >> and
> > > >>>>> introduce the practice of true unit-tests. In the new
baseline at
> > the
> > > >>>>> beginning there will be no ability to run an end-to-end
scenario,
> > > >> thus
> > > >>> we
> > > >>>>> will be forced to write unit-tests. So far, such practice
was
> hard
> > to
> > > >>>>> implement because of tight coupling between Ignite components
and
> > > >>> inability
> > > >>>>> to instantiate components without an instance of KernalContext.
> For
> > > >>>>> example, we should be able to thoroughly test internal
> primitives,
> > > >>> such as
> > > >>>>> replication protocol (without actual communication), distributed
> > > >>>>> metastorage contracts, persistence layer, etc.
> > > >>>>> - We will significantly reduce the development cycle in
the
> > beginning
> > > >>>>> (right now the RunAll takes two hours of astronomical
time with
> > empty
> > > >>> TC;
> > > >>>>> in the new approach developer will be able to run ALL
tests
> locally
> > > >> in
> > > >>> a
> > > >>>>> matter of minutes)
> > > >>>>> - We can get rid of TC bot and enforce green TC by integrating
TC
> > > >> build
> > > >>>>> results with GitHub PRs (the same way Travis is currently
> > integrated
> > > >>> to PR
> > > >>>>> check). We should restrict PR merge without a TC check
> > > >>>>> - We will still have to re-write all tests, but only once.
If we
> > try
> > > >> to
> > > >>>>> modify the old codebase, we would need to modify all the
tests
> for
> > > >>> every
> > > >>>>> major change (public API change, configuration change)
> > > >>>>> - We will have fewer conflicts when working together.
For
> example,
> > I
> > > >>>>> cannot imagine how one would merge two changes of getting
rid of
> > > >>>>> IgniteFuture and changes in replication protocol, for
example
> > > >>>>>
> > > >>>>> Technically, I would suggest creating a new repository
for Ignite
> > 3.0
> > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer
to me)
> > and a
> > > >>> new
> > > >>>>> Ignite 3.0 TeamCity project.
> > > >>>>>
> > > >>>>> While it may seem quite radical, I do believe that this
approach
> > will
> > > >>> give
> > > >>>>> us more benefits than trying to make such major changes
in the
> > > >> existing
> > > >>>>> codebase. If needed, let's schedule a discord chat like
before to
> > > >>> discuss
> > > >>>>> this.
> > > >>>>>
> > > >>>>> WDYT?
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>


-- 
Best regards,
Andrey V. Mashenkov

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message