ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kseniya Romanova <romanova.ks....@gmail.com>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Tue, 03 Nov 2020 10:31:50 GMT
Let's do this community discussion open. Here's the link on zoom call in
Russian for Friday 6 PM:
https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/

вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhikov@apache.org>:

> Time works for me.
>
> > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <alexey.goncharuk@gmail.com>
> написал(а):
> >
> > Nikolay,
> >
> > I am up for the call. I will try to explain my reasoning in greater
> detail
> > and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
> >
> > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhikov@apache.org>:
> >
> >> Igniters, should we have a call for this topic?
> >>
> >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org>
> >> написал(а):
> >>>
> >>>> not intend to rewrite everything from scratch
> >>>
> >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> >>>> regardless of how we choose to proceed.
> >>>
> >>> Alexey, thank you for the explanation, this addresses all of my
> concerns.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> >> andrey.mashenkov@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi, Igniters.
> >>>>
> >>>> * AFAIU, we need a new repo if we want to apply different restrictions
> >> to
> >>>> pull requests,
> >>>> otherwise I see no difference for myself.
> >>>> E.g. make static analysis (do we have?), compile, styles, and javadoc
> >>>> checks mandatory.
> >>>>
> >>>> I think that relaxed requirements here will lead to bad product
> quality.
> >>>>
> >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> >>>> During active development tests will be broken most of time, so,
> >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> >> featured
> >>>> environment to run them and of course make test's code clear and avoid
> >>>> bad/non-relevant ones.
> >>>>
> >>>> * I like bottom-up approach.
> >>>> With it we could make a better framework. I mean clear component
> >> lifecycle,
> >>>> component wiring mechanics, general methods to approach core
> components
> >>>> such as exchange/communication
> >>>> to avoid code mess like we have in ExchangeFuture with all these
> custom
> >>>> callbacks for each component, interfaces like
> >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> >>>> a pack of
> >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> >>>> and so on in various unexpected places.
> >>>> Hope, we will be able to port most of the good code to the new
> framework
> >>>> version.
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> >>>> alexey.goncharuk@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Nikolay, Pavel,
> >>>>>
> >>>>> Thanks for the feedback! First of all, I wanted to stress that I
do
> not
> >>>>> intend to rewrite everything from scratch (I never used this phrase).
> >>>> There
> >>>>> are significant parts of code that would be moved with minimal
> >>>>> modifications. Second, I never said that we will get rid of the
old
> >> tests
> >>>>> codebase. Every single test from Ignite 2.x should be moved to
> Ignite 3
> >>>>> regardless of how we choose to proceed.
> >>>>>
> >>>>> My point is that for some parts of the code a clean bottom-up
> >>>>> implementation will be cheaper in many ways. Let me give you a few
> >>>> concrete
> >>>>> examples:
> >>>>>
> >>>>>  - I think no one can object that we need a cleanly separated
> >>>> persistence
> >>>>>  layer for Ignite. There is a very raw draft IEP for this already.
On
> >>>> the
> >>>>>  other hand, I think we also can agree that we need a split-brain
> >>>>> resistant
> >>>>>  replication protocol for caches. There is also an IEP for this.
> >>>> Neither
> >>>>> of
> >>>>>  the changes is a good fit for 2.x because they are likely to
> >> introduce
> >>>>>  breaking changes in the persistence layer, configuration and
> >> behavior.
> >>>>>  Additionally, these components are now tightly coupled, so there
is
> >> no
> >>>>> way
> >>>>>  these two changes can be implemented in parallel and then merged
> >>>>> together
> >>>>>  easily. So what we will end up with is having to implement these
> >>>> changes
> >>>>>  sequentially, fixing all existing tests twice, and essentially
> >>>> throwing
> >>>>>  away half of the work done because the other part of the change
is
> >>>>>  re-implemented
> >>>>>  - Similar example goes with getting rid of IgniteInternalFuture
and
> >>>>>  replacing it with CompletableFuture, and any other change that
> >> touches
> >>>>> the
> >>>>>  asynchronous part of the code.
> >>>>>
> >>>>> Third, I do not see how this choice affects the UX of Ignite. The
end
> >>>> user
> >>>>> experience must be fixed in the IEP regardless of the development
> >> process
> >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> confirms
> >>>>> that.
> >>>>>
> >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> >>>> reformulate,
> >>>>> my point is that we might agree to have a single development master
> >>>> branch
> >>>>> with 'disassembled' end-to-end functionality for some period of
time
> to
> >>>>> speed up development, and re-assemble the core features after having
> >>>>> submodules tested independently.
> >>>>>
> >>>>> Nikolay,
> >>>>>> We have many features that have to evolve.
> >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
> >>>>> This is not very specific. In the end, resources are limited and
we
> >> will
> >>>>> not be able to drive both tracks simultaneously, especially after
a
> >>>> couple
> >>>>> of features having been implemented for Ignite 3.0. If there are
> indeed
> >>>>> some major changes that we want to do in Ignite 2.x instead of
> putting
> >>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
> >> that's
> >>>>> why I am eager to move forward with Ignite 3.0.
> >>>>>
> >>>>>> We have bugs and issues that can be fixed in 2.x without breaking
> >>>> backward
> >>>>> compatibility.
> >>>>>> We have many users who are happy with the 2.x with all it’s
issues.
> >>>>> These changes will be covered by end-to-end tests and migrated to
> >> Ignite
> >>>>> 3.0, so I see no issues here.
> >>>>>
> >>>>> Finally, Anton & Nikolay
> >>>>> I do not have an estimate for this simply because the activity is
> >>>>> community-driven and it depends on the number of people willing
to
> >>>>> contribute. With the current pace, I would hope to have an RC of
> Ignite
> >>>> 3.0
> >>>>> to be ready by the end of 2021. My gut feeling is that by moving
with
> >>>>> incremental changes, we will not be able to implement even half
of
> the
> >>>>> wishlist by that time.
> >>>>> I doubt that releasing several major releases with breaking changes
> >> will
> >>>>> make Ignite users happy either because each upgrade will cost Ignite
> >>>> users
> >>>>> money, so the fewer major versions we release, the better. Thus
my
> wish
> >>>> to
> >>>>> include all breaking changes in one release.
> >>>>>
> >>>>> I'll be now quiet for a while, let's see what other community members
> >>>>> think.
> >>>>>
> >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org>:
> >>>>>
> >>>>>> 1. Rewriting from scratch is never a good idea.
> >>>>>> We don't want to follow the path of Netscape and lose all our
users
> >>>>>> by the time we have a working 3.0 [1]
> >>>>>>
> >>>>>> 2. Not sure about new repo - seems like some pain and no gain,
> what's
> >>>> the
> >>>>>> problem with a branch?
> >>>>>>
> >>>>>> 3. We should keep existing integration tests when possible.
> >>>>>> We have accumulated a lot of edge case knowledge over the years,
> >>>>>> it is not a good idea to send all of that down the drain.
> >>>>>> Yes, integration tests are slow, but they are the most valuable.
> >>>>>> I think we can move more stuff into nightly runs and have a
fast and
> >>>>> modern
> >>>>>> basic suite.
> >>>>>>
> >>>>>>
> >>>>>> Alexey, you are much more familiar with the Ignite core codebase
> than
> >>>>> most
> >>>>>> of us,
> >>>>>> can you please explain in more detail which particular feature,
in
> >> your
> >>>>>> opinion,
> >>>>>> mandates this "start from scratch" approach?
> >>>>>> Is it really not possible at all to follow a less radical way?
> >>>>>>
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org
> >
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello, Alexey.
> >>>>>>>
> >>>>>>> I think that «rewriting from scratch» approach has a high
risk to
> >>>> make
> >>>>>> new
> >>>>>>> features unusable.
> >>>>>>> At the time Ignite2 was started no-one wants to do bad UX
or bad
> >>>>>> features.
> >>>>>>> Nevertheless, it happen.
> >>>>>>>
> >>>>>>> I think we can avoid it with the Ignite3 and successors
if we will
> >>>> move
> >>>>>>> step by step without keeping backward compatibility
> >>>>>>> With the step by step approach, we can focus on each component
> >>>>>> separately.
> >>>>>>>
> >>>>>>>> What new features are we planning to implement for Ignite
2.x?
> >>>>>>>
> >>>>>>> We have many features that have to evolve.
> >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
etc.
> >>>>>>> We have bugs and issues that can be fixed in 2.x without
breaking
> >>>>>> backward
> >>>>>>> compatibility.
> >>>>>>> We have many users who are happy with the 2.x with all it’s
issues.
> >>>>>>>
> >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org>
> >>>>> написал(а):
> >>>>>>>>
> >>>>>>>> Alexey,
> >>>>>>>>
> >>>>>>>> Do we have any estimates of how fast we'll be able to
gain
> >>>>>>> production-ready
> >>>>>>>> AI 3.0 in case of a "new repo" choice?
> >>>>>>>>
> >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
> >>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Nikolay,
> >>>>>>>>>
> >>>>>>>>> What new features are we planning to implement for
Ignite 2.x? I
> >>>>> think
> >>>>>>> once
> >>>>>>>>> we commence working on Ignite 3.0, we should gradually
cease the
> >>>>>>> activity
> >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
development
> >>>>> will
> >>>>>> be
> >>>>>>>>> overwhelming regardless of how we choose to proceed.
> >>>>>>>>>
> >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov
<
> nizhikov@apache.org
> >>>>> :
> >>>>>>>>>
> >>>>>>>>>> To be clear:
> >>>>>>>>>>
> >>>>>>>>>>> I would suggest creating a new repository
for Ignite 3.0
> >>>>> (perhaps, a
> >>>>>>>>> new
> >>>>>>>>>> clean branch, but a new repo looks nicer to
me) and a new Ignite
> >>>>> 3.0
> >>>>>>>>>> TeamCity project.
> >>>>>>>>>>
> >>>>>>>>>> +1 for new Team City project.
> >>>>>>>>>> +1 for new branch for Ignite3.
> >>>>>>>>>> -1 for new repo.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay
Izhikov <
> >>>> NIzhikov.dev@gmail.com
> >>>>>>
> >>>>>>>>>> написал(а):
> >>>>>>>>>>>
> >>>>>>>>>>> Hello, Alexey.
> >>>>>>>>>>>
> >>>>>>>>>>> I think it will hurt our project more than
help.
> >>>>>>>>>>> Developing new features for 2 separate branches
with the
> >>>> different
> >>>>>>> APIs
> >>>>>>>>>> and internal structure is overwhelming
> >>>>>>>>>>>
> >>>>>>>>>>> Maybe we should relax a bit requirements
for Ignite3?
> >>>>>>>>>>> Maybe we should move step by step and make
Ignite3 with new
> >>>>>>>>>> configuration than Ignite4 with new transactions,
etc?
> >>>>>>>>>>>
> >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey
Goncharuk <
> >>>>>>>>> alexey.goncharuk@gmail.com>
> >>>>>>>>>> написал(а):
> >>>>>>>>>>>>
> >>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I wanted to pitch a rather radical idea
regarding the Ignite
> >>>> 3.0
> >>>>>>>>>>>> development which has occurred to me
some time ago.
> >>>>>>>>>>>>
> >>>>>>>>>>>> We already have several IEPs targeted
to Ignite 3.0 which
> imply
> >>>>>> major
> >>>>>>>>>>>> changes to the codebase (the change
in replication protocol
> and
> >>>>>> thus
> >>>>>>>>>>>> transactions, change in binary format,
updated metastorage,
> >>>> etc).
> >>>>>> We
> >>>>>>>>>> also
> >>>>>>>>>>>> planned significant changes in public
APIs: configuration
> >>>> format
> >>>>>>>>> change,
> >>>>>>>>>>>> improvements in cache APIs, SQL APIs,
transaction mode rework.
> >>>>> The
> >>>>>>>>>> wishlist
> >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So, I was wondering whether it makes
sense to try to change
> the
> >>>>> old
> >>>>>>>>>>>> codebase, or start with a new baseline
and move old pieces of
> >>>>> code
> >>>>>>>>> that
> >>>>>>>>>> do
> >>>>>>>>>>>> not require significant rework. Personally,
I would go with
> the
> >>>>>>> second
> >>>>>>>>>>>> option for the following reasons:
> >>>>>>>>>>>>
> >>>>>>>>>>>> - We have a chance to shift the development
paradigm in the
> >>>>> project
> >>>>>>>>> and
> >>>>>>>>>>>> introduce the practice of true unit-tests.
In the new baseline
> >>>> at
> >>>>>> the
> >>>>>>>>>>>> beginning there will be no ability to
run an end-to-end
> >>>> scenario,
> >>>>>>>>> thus
> >>>>>>>>>> we
> >>>>>>>>>>>> will be forced to write unit-tests.
So far, such practice was
> >>>>> hard
> >>>>>> to
> >>>>>>>>>>>> implement because of tight coupling
between Ignite components
> >>>> and
> >>>>>>>>>> inability
> >>>>>>>>>>>> to instantiate components without an
instance of
> KernalContext.
> >>>>> For
> >>>>>>>>>>>> example, we should be able to thoroughly
test internal
> >>>>> primitives,
> >>>>>>>>>> such as
> >>>>>>>>>>>> replication protocol (without actual
communication),
> >>>> distributed
> >>>>>>>>>>>> metastorage contracts, persistence layer,
etc.
> >>>>>>>>>>>> - We will significantly reduce the development
cycle in the
> >>>>>> beginning
> >>>>>>>>>>>> (right now the RunAll takes two hours
of astronomical time
> with
> >>>>>> empty
> >>>>>>>>>> TC;
> >>>>>>>>>>>> in the new approach developer will be
able to run ALL tests
> >>>>> locally
> >>>>>>>>> in
> >>>>>>>>>> a
> >>>>>>>>>>>> matter of minutes)
> >>>>>>>>>>>> - We can get rid of TC bot and enforce
green TC by integrating
> >>>> TC
> >>>>>>>>> build
> >>>>>>>>>>>> results with GitHub PRs (the same way
Travis is currently
> >>>>>> integrated
> >>>>>>>>>> to PR
> >>>>>>>>>>>> check). We should restrict PR merge
without a TC check
> >>>>>>>>>>>> - We will still have to re-write all
tests, but only once. If
> >>>> we
> >>>>>> try
> >>>>>>>>> to
> >>>>>>>>>>>> modify the old codebase, we would need
to modify all the tests
> >>>>> for
> >>>>>>>>>> every
> >>>>>>>>>>>> major change (public API change, configuration
change)
> >>>>>>>>>>>> - We will have fewer conflicts when
working together. For
> >>>>> example,
> >>>>>> I
> >>>>>>>>>>>> cannot imagine how one would merge two
changes of getting rid
> >>>> of
> >>>>>>>>>>>> IgniteFuture and changes in replication
protocol, for example
> >>>>>>>>>>>>
> >>>>>>>>>>>> Technically, I would suggest creating
a new repository for
> >>>> Ignite
> >>>>>> 3.0
> >>>>>>>>>>>> (perhaps, a new clean branch, but a
new repo looks nicer to
> me)
> >>>>>> and a
> >>>>>>>>>> new
> >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> >>>>>>>>>>>>
> >>>>>>>>>>>> While it may seem quite radical, I do
believe that this
> >>>> approach
> >>>>>> will
> >>>>>>>>>> give
> >>>>>>>>>>>> us more benefits than trying to make
such major changes in the
> >>>>>>>>> existing
> >>>>>>>>>>>> codebase. If needed, let's schedule
a discord chat like before
> >>>> to
> >>>>>>>>>> discuss
> >>>>>>>>>>>> this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> WDYT?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrey V. Mashenkov
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message