ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Goncharuk <alexey.goncha...@gmail.com>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Mon, 02 Nov 2020 15:18:27 GMT
Nikolay, Pavel,

Thanks for the feedback! First of all, I wanted to stress that I do not
intend to rewrite everything from scratch (I never used this phrase). There
are significant parts of code that would be moved with minimal
modifications. Second, I never said that we will get rid of the old tests
codebase. Every single test from Ignite 2.x should be moved to Ignite 3
regardless of how we choose to proceed.

My point is that for some parts of the code a clean bottom-up
implementation will be cheaper in many ways. Let me give you a few concrete
examples:

   - I think no one can object that we need a cleanly separated persistence
   layer for Ignite. There is a very raw draft IEP for this already. On the
   other hand, I think we also can agree that we need a split-brain resistant
   replication protocol for caches. There is also an IEP for this. Neither of
   the changes is a good fit for 2.x because they are likely to introduce
   breaking changes in the persistence layer, configuration and behavior.
   Additionally, these components are now tightly coupled, so there is no way
   these two changes can be implemented in parallel and then merged together
   easily. So what we will end up with is having to implement these changes
   sequentially, fixing all existing tests twice, and essentially throwing
   away half of the work done because the other part of the change is
   re-implemented
   - Similar example goes with getting rid of IgniteInternalFuture and
   replacing it with CompletableFuture, and any other change that touches the
   asynchronous part of the code.

Third, I do not see how this choice affects the UX of Ignite. The end user
experience must be fixed in the IEP regardless of the development process
and the fact that we have gaps in this area in Ignite 2.x just confirms
that.

Pavel, agree that a repo/branch is a technicality, I guess if reformulate,
my point is that we might agree to have a single development master branch
with 'disassembled' end-to-end functionality for some period of time to
speed up development, and re-assemble the core features after having
submodules tested independently.

Nikolay,
  >We have many features that have to evolve.
  >Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
This is not very specific. In the end, resources are limited and we will
not be able to drive both tracks simultaneously, especially after a couple
of features having been implemented for Ignite 3.0. If there are indeed
some major changes that we want to do in Ignite 2.x instead of putting
effort into 3.0 - let's discuss them. I am just not aware of any, that's
why I am eager to move forward with Ignite 3.0.

>We have bugs and issues that can be fixed in 2.x without breaking backward
compatibility.
>We have many users who are happy with the 2.x with all it’s issues.
These changes will be covered by end-to-end tests and migrated to Ignite
3.0, so I see no issues here.

Finally, Anton & Nikolay
I do not have an estimate for this simply because the activity is
community-driven and it depends on the number of people willing to
contribute. With the current pace, I would hope to have an RC of Ignite 3.0
to be ready by the end of 2021. My gut feeling is that by moving with
incremental changes, we will not be able to implement even half of the
wishlist by that time.
I doubt that releasing several major releases with breaking changes will
make Ignite users happy either because each upgrade will cost Ignite users
money, so the fewer major versions we release, the better. Thus my wish to
include all breaking changes in one release.

I'll be now quiet for a while, let's see what other community members think.

пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org>:

> 1. Rewriting from scratch is never a good idea.
> We don't want to follow the path of Netscape and lose all our users
> by the time we have a working 3.0 [1]
>
> 2. Not sure about new repo - seems like some pain and no gain, what's the
> problem with a branch?
>
> 3. We should keep existing integration tests when possible.
> We have accumulated a lot of edge case knowledge over the years,
> it is not a good idea to send all of that down the drain.
> Yes, integration tests are slow, but they are the most valuable.
> I think we can move more stuff into nightly runs and have a fast and modern
> basic suite.
>
>
> Alexey, you are much more familiar with the Ignite core codebase than most
> of us,
> can you please explain in more detail which particular feature, in your
> opinion,
> mandates this "start from scratch" approach?
> Is it really not possible at all to follow a less radical way?
>
>
> [1]
>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>
> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org>
> wrote:
>
> > Hello, Alexey.
> >
> > I think that «rewriting from scratch» approach has a high risk to make
> new
> > features unusable.
> > At the time Ignite2 was started no-one wants to do bad UX or bad
> features.
> > Nevertheless, it happen.
> >
> > I think we can avoid it with the Ignite3 and successors if we will move
> > step by step without keeping backward compatibility
> > With the step by step approach, we can focus on each component
> separately.
> >
> > > What new features are we planning to implement for Ignite 2.x?
> >
> > We have many features that have to evolve.
> > Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> > We have bugs and issues that can be fixed in 2.x without breaking
> backward
> > compatibility.
> > We have many users who are happy with the 2.x with all it’s issues.
> >
> > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org> написал(а):
> > >
> > > Alexey,
> > >
> > > Do we have any estimates of how fast we'll be able to gain
> > production-ready
> > > AI 3.0 in case of a "new repo" choice?
> > >
> > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> > alexey.goncharuk@gmail.com>
> > > wrote:
> > >
> > >> Nikolay,
> > >>
> > >> What new features are we planning to implement for Ignite 2.x? I think
> > once
> > >> we commence working on Ignite 3.0, we should gradually cease the
> > activity
> > >> on Ignite 2.x to mere bugfixes because such parallel development will
> be
> > >> overwhelming regardless of how we choose to proceed.
> > >>
> > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org>:
> > >>
> > >>> To be clear:
> > >>>
> > >>>> I would suggest creating a new repository for Ignite 3.0 (perhaps,
a
> > >> new
> > >>> clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> > >>> TeamCity project.
> > >>>
> > >>> +1 for new Team City project.
> > >>> +1 for new branch for Ignite3.
> > >>> -1 for new repo.
> > >>>
> > >>>
> > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <NIzhikov.dev@gmail.com>
> > >>> написал(а):
> > >>>>
> > >>>> Hello, Alexey.
> > >>>>
> > >>>> I think it will hurt our project more than help.
> > >>>> Developing new features for 2 separate branches with the different
> > APIs
> > >>> and internal structure is overwhelming
> > >>>>
> > >>>> Maybe we should relax a bit requirements for Ignite3?
> > >>>> Maybe we should move step by step and make Ignite3 with new
> > >>> configuration than Ignite4 with new transactions, etc?
> > >>>>
> > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> > >> alexey.goncharuk@gmail.com>
> > >>> написал(а):
> > >>>>>
> > >>>>> Igniters,
> > >>>>>
> > >>>>> I wanted to pitch a rather radical idea regarding the Ignite
3.0
> > >>>>> development which has occurred to me some time ago.
> > >>>>>
> > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply
> major
> > >>>>> changes to the codebase (the change in replication protocol
and
> thus
> > >>>>> transactions, change in binary format, updated metastorage,
etc).
> We
> > >>> also
> > >>>>> planned significant changes in public APIs: configuration format
> > >> change,
> > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
The
> > >>> wishlist
> > >>>>> of changes for Ignite 3.0 is huge.
> > >>>>>
> > >>>>> So, I was wondering whether it makes sense to try to change
the old
> > >>>>> codebase, or start with a new baseline and move old pieces
of code
> > >> that
> > >>> do
> > >>>>> not require significant rework. Personally, I would go with
the
> > second
> > >>>>> option for the following reasons:
> > >>>>>
> > >>>>> - We have a chance to shift the development paradigm in the
project
> > >> and
> > >>>>> introduce the practice of true unit-tests. In the new baseline
at
> the
> > >>>>> beginning there will be no ability to run an end-to-end scenario,
> > >> thus
> > >>> we
> > >>>>> will be forced to write unit-tests. So far, such practice was
hard
> to
> > >>>>> implement because of tight coupling between Ignite components
and
> > >>> inability
> > >>>>> to instantiate components without an instance of KernalContext.
For
> > >>>>> example, we should be able to thoroughly test internal primitives,
> > >>> such as
> > >>>>> replication protocol (without actual communication), distributed
> > >>>>> metastorage contracts, persistence layer, etc.
> > >>>>> - We will significantly reduce the development cycle in the
> beginning
> > >>>>> (right now the RunAll takes two hours of astronomical time
with
> empty
> > >>> TC;
> > >>>>> in the new approach developer will be able to run ALL tests
locally
> > >> in
> > >>> a
> > >>>>> matter of minutes)
> > >>>>> - We can get rid of TC bot and enforce green TC by integrating
TC
> > >> build
> > >>>>> results with GitHub PRs (the same way Travis is currently
> integrated
> > >>> to PR
> > >>>>> check). We should restrict PR merge without a TC check
> > >>>>> - We will still have to re-write all tests, but only once.
If we
> try
> > >> to
> > >>>>> modify the old codebase, we would need to modify all the tests
for
> > >>> every
> > >>>>> major change (public API change, configuration change)
> > >>>>> - We will have fewer conflicts when working together. For example,
> I
> > >>>>> cannot imagine how one would merge two changes of getting rid
of
> > >>>>> IgniteFuture and changes in replication protocol, for example
> > >>>>>
> > >>>>> Technically, I would suggest creating a new repository for
Ignite
> 3.0
> > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to
me)
> and a
> > >>> new
> > >>>>> Ignite 3.0 TeamCity project.
> > >>>>>
> > >>>>> While it may seem quite radical, I do believe that this approach
> will
> > >>> give
> > >>>>> us more benefits than trying to make such major changes in
the
> > >> existing
> > >>>>> codebase. If needed, let's schedule a discord chat like before
to
> > >>> discuss
> > >>>>> this.
> > >>>>>
> > >>>>> WDYT?
> > >>>>
> > >>>
> > >>>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message