ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Tupitsyn <ptupit...@apache.org>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Tue, 03 Nov 2020 10:51:46 GMT
Kseniya,

Thanks for scheduling this call.
Do you think we can switch to English if non-Russian speaking community
members decide to join?

On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <romanova.ks.spb@gmail.com>
wrote:

> Let's do this community discussion open. Here's the link on zoom call in
> Russian for Friday 6 PM:
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
>
> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhikov@apache.org>:
>
> > Time works for me.
> >
> > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <alexey.goncharuk@gmail.com
> >
> > написал(а):
> > >
> > > Nikolay,
> > >
> > > I am up for the call. I will try to explain my reasoning in greater
> > detail
> > > and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
> > >
> > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhikov@apache.org>:
> > >
> > >> Igniters, should we have a call for this topic?
> > >>
> > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org>
> > >> написал(а):
> > >>>
> > >>>> not intend to rewrite everything from scratch
> > >>>
> > >>>> Every single test from Ignite 2.x should be moved to Ignite 3
> > >>>> regardless of how we choose to proceed.
> > >>>
> > >>> Alexey, thank you for the explanation, this addresses all of my
> > concerns.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
> > >> andrey.mashenkov@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi, Igniters.
> > >>>>
> > >>>> * AFAIU, we need a new repo if we want to apply different
> restrictions
> > >> to
> > >>>> pull requests,
> > >>>> otherwise I see no difference for myself.
> > >>>> E.g. make static analysis (do we have?), compile, styles, and
> javadoc
> > >>>> checks mandatory.
> > >>>>
> > >>>> I think that relaxed requirements here will lead to bad product
> > quality.
> > >>>>
> > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
> > >>>> During active development tests will be broken most of time, so,
> > >>>> I'd port them e.g. suite-by-suite once we will have a stable and
> > >> featured
> > >>>> environment to run them and of course make test's code clear and
> avoid
> > >>>> bad/non-relevant ones.
> > >>>>
> > >>>> * I like bottom-up approach.
> > >>>> With it we could make a better framework. I mean clear component
> > >> lifecycle,
> > >>>> component wiring mechanics, general methods to approach core
> > components
> > >>>> such as exchange/communication
> > >>>> to avoid code mess like we have in ExchangeFuture with all these
> > custom
> > >>>> callbacks for each component, interfaces like
> > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
> > >>>> a pack of
> > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
> > >>>> and so on in various unexpected places.
> > >>>> Hope, we will be able to port most of the good code to the new
> > framework
> > >>>> version.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
> > >>>> alexey.goncharuk@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Nikolay, Pavel,
> > >>>>>
> > >>>>> Thanks for the feedback! First of all, I wanted to stress that
I do
> > not
> > >>>>> intend to rewrite everything from scratch (I never used this
> phrase).
> > >>>> There
> > >>>>> are significant parts of code that would be moved with minimal
> > >>>>> modifications. Second, I never said that we will get rid of
the old
> > >> tests
> > >>>>> codebase. Every single test from Ignite 2.x should be moved
to
> > Ignite 3
> > >>>>> regardless of how we choose to proceed.
> > >>>>>
> > >>>>> My point is that for some parts of the code a clean bottom-up
> > >>>>> implementation will be cheaper in many ways. Let me give you
a few
> > >>>> concrete
> > >>>>> examples:
> > >>>>>
> > >>>>>  - I think no one can object that we need a cleanly separated
> > >>>> persistence
> > >>>>>  layer for Ignite. There is a very raw draft IEP for this already.
> On
> > >>>> the
> > >>>>>  other hand, I think we also can agree that we need a split-brain
> > >>>>> resistant
> > >>>>>  replication protocol for caches. There is also an IEP for
this.
> > >>>> Neither
> > >>>>> of
> > >>>>>  the changes is a good fit for 2.x because they are likely
to
> > >> introduce
> > >>>>>  breaking changes in the persistence layer, configuration and
> > >> behavior.
> > >>>>>  Additionally, these components are now tightly coupled, so
there
> is
> > >> no
> > >>>>> way
> > >>>>>  these two changes can be implemented in parallel and then
merged
> > >>>>> together
> > >>>>>  easily. So what we will end up with is having to implement
these
> > >>>> changes
> > >>>>>  sequentially, fixing all existing tests twice, and essentially
> > >>>> throwing
> > >>>>>  away half of the work done because the other part of the change
is
> > >>>>>  re-implemented
> > >>>>>  - Similar example goes with getting rid of IgniteInternalFuture
> and
> > >>>>>  replacing it with CompletableFuture, and any other change
that
> > >> touches
> > >>>>> the
> > >>>>>  asynchronous part of the code.
> > >>>>>
> > >>>>> Third, I do not see how this choice affects the UX of Ignite.
The
> end
> > >>>> user
> > >>>>> experience must be fixed in the IEP regardless of the development
> > >> process
> > >>>>> and the fact that we have gaps in this area in Ignite 2.x just
> > confirms
> > >>>>> that.
> > >>>>>
> > >>>>> Pavel, agree that a repo/branch is a technicality, I guess
if
> > >>>> reformulate,
> > >>>>> my point is that we might agree to have a single development
master
> > >>>> branch
> > >>>>> with 'disassembled' end-to-end functionality for some period
of
> time
> > to
> > >>>>> speed up development, and re-assemble the core features after
> having
> > >>>>> submodules tested independently.
> > >>>>>
> > >>>>> Nikolay,
> > >>>>>> We have many features that have to evolve.
> > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
etc.
> > >>>>> This is not very specific. In the end, resources are limited
and we
> > >> will
> > >>>>> not be able to drive both tracks simultaneously, especially
after a
> > >>>> couple
> > >>>>> of features having been implemented for Ignite 3.0. If there
are
> > indeed
> > >>>>> some major changes that we want to do in Ignite 2.x instead
of
> > putting
> > >>>>> effort into 3.0 - let's discuss them. I am just not aware of
any,
> > >> that's
> > >>>>> why I am eager to move forward with Ignite 3.0.
> > >>>>>
> > >>>>>> We have bugs and issues that can be fixed in 2.x without
breaking
> > >>>> backward
> > >>>>> compatibility.
> > >>>>>> We have many users who are happy with the 2.x with all
it’s
> issues.
> > >>>>> These changes will be covered by end-to-end tests and migrated
to
> > >> Ignite
> > >>>>> 3.0, so I see no issues here.
> > >>>>>
> > >>>>> Finally, Anton & Nikolay
> > >>>>> I do not have an estimate for this simply because the activity
is
> > >>>>> community-driven and it depends on the number of people willing
to
> > >>>>> contribute. With the current pace, I would hope to have an
RC of
> > Ignite
> > >>>> 3.0
> > >>>>> to be ready by the end of 2021. My gut feeling is that by moving
> with
> > >>>>> incremental changes, we will not be able to implement even
half of
> > the
> > >>>>> wishlist by that time.
> > >>>>> I doubt that releasing several major releases with breaking
changes
> > >> will
> > >>>>> make Ignite users happy either because each upgrade will cost
> Ignite
> > >>>> users
> > >>>>> money, so the fewer major versions we release, the better.
Thus my
> > wish
> > >>>> to
> > >>>>> include all breaking changes in one release.
> > >>>>>
> > >>>>> I'll be now quiet for a while, let's see what other community
> members
> > >>>>> think.
> > >>>>>
> > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org
> >:
> > >>>>>
> > >>>>>> 1. Rewriting from scratch is never a good idea.
> > >>>>>> We don't want to follow the path of Netscape and lose all
our
> users
> > >>>>>> by the time we have a working 3.0 [1]
> > >>>>>>
> > >>>>>> 2. Not sure about new repo - seems like some pain and no
gain,
> > what's
> > >>>> the
> > >>>>>> problem with a branch?
> > >>>>>>
> > >>>>>> 3. We should keep existing integration tests when possible.
> > >>>>>> We have accumulated a lot of edge case knowledge over the
years,
> > >>>>>> it is not a good idea to send all of that down the drain.
> > >>>>>> Yes, integration tests are slow, but they are the most
valuable.
> > >>>>>> I think we can move more stuff into nightly runs and have
a fast
> and
> > >>>>> modern
> > >>>>>> basic suite.
> > >>>>>>
> > >>>>>>
> > >>>>>> Alexey, you are much more familiar with the Ignite core
codebase
> > than
> > >>>>> most
> > >>>>>> of us,
> > >>>>>> can you please explain in more detail which particular
feature, in
> > >> your
> > >>>>>> opinion,
> > >>>>>> mandates this "start from scratch" approach?
> > >>>>>> Is it really not possible at all to follow a less radical
way?
> > >>>>>>
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
> > >>>>>>
> > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <
> nizhikov@apache.org
> > >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hello, Alexey.
> > >>>>>>>
> > >>>>>>> I think that «rewriting from scratch» approach has
a high risk to
> > >>>> make
> > >>>>>> new
> > >>>>>>> features unusable.
> > >>>>>>> At the time Ignite2 was started no-one wants to do
bad UX or bad
> > >>>>>> features.
> > >>>>>>> Nevertheless, it happen.
> > >>>>>>>
> > >>>>>>> I think we can avoid it with the Ignite3 and successors
if we
> will
> > >>>> move
> > >>>>>>> step by step without keeping backward compatibility
> > >>>>>>> With the step by step approach, we can focus on each
component
> > >>>>>> separately.
> > >>>>>>>
> > >>>>>>>> What new features are we planning to implement
for Ignite 2.x?
> > >>>>>>>
> > >>>>>>> We have many features that have to evolve.
> > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
etc.
> > >>>>>>> We have bugs and issues that can be fixed in 2.x without
breaking
> > >>>>>> backward
> > >>>>>>> compatibility.
> > >>>>>>> We have many users who are happy with the 2.x with
all it’s
> issues.
> > >>>>>>>
> > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov
<av@apache.org>
> > >>>>> написал(а):
> > >>>>>>>>
> > >>>>>>>> Alexey,
> > >>>>>>>>
> > >>>>>>>> Do we have any estimates of how fast we'll be able
to gain
> > >>>>>>> production-ready
> > >>>>>>>> AI 3.0 in case of a "new repo" choice?
> > >>>>>>>>
> > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk
<
> > >>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Nikolay,
> > >>>>>>>>>
> > >>>>>>>>> What new features are we planning to implement
for Ignite 2.x?
> I
> > >>>>> think
> > >>>>>>> once
> > >>>>>>>>> we commence working on Ignite 3.0, we should
gradually cease
> the
> > >>>>>>> activity
> > >>>>>>>>> on Ignite 2.x to mere bugfixes because such
parallel
> development
> > >>>>> will
> > >>>>>> be
> > >>>>>>>>> overwhelming regardless of how we choose to
proceed.
> > >>>>>>>>>
> > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay
Izhikov <
> > nizhikov@apache.org
> > >>>>> :
> > >>>>>>>>>
> > >>>>>>>>>> To be clear:
> > >>>>>>>>>>
> > >>>>>>>>>>> I would suggest creating a new repository
for Ignite 3.0
> > >>>>> (perhaps, a
> > >>>>>>>>> new
> > >>>>>>>>>> clean branch, but a new repo looks nicer
to me) and a new
> Ignite
> > >>>>> 3.0
> > >>>>>>>>>> TeamCity project.
> > >>>>>>>>>>
> > >>>>>>>>>> +1 for new Team City project.
> > >>>>>>>>>> +1 for new branch for Ignite3.
> > >>>>>>>>>> -1 for new repo.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay
Izhikov <
> > >>>> NIzhikov.dev@gmail.com
> > >>>>>>
> > >>>>>>>>>> написал(а):
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hello, Alexey.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think it will hurt our project more
than help.
> > >>>>>>>>>>> Developing new features for 2 separate
branches with the
> > >>>> different
> > >>>>>>> APIs
> > >>>>>>>>>> and internal structure is overwhelming
> > >>>>>>>>>>>
> > >>>>>>>>>>> Maybe we should relax a bit requirements
for Ignite3?
> > >>>>>>>>>>> Maybe we should move step by step and
make Ignite3 with new
> > >>>>>>>>>> configuration than Ignite4 with new transactions,
etc?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14,
Alexey Goncharuk <
> > >>>>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>>>> написал(а):
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I wanted to pitch a rather radical
idea regarding the Ignite
> > >>>> 3.0
> > >>>>>>>>>>>> development which has occurred
to me some time ago.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We already have several IEPs targeted
to Ignite 3.0 which
> > imply
> > >>>>>> major
> > >>>>>>>>>>>> changes to the codebase (the change
in replication protocol
> > and
> > >>>>>> thus
> > >>>>>>>>>>>> transactions, change in binary
format, updated metastorage,
> > >>>> etc).
> > >>>>>> We
> > >>>>>>>>>> also
> > >>>>>>>>>>>> planned significant changes in
public APIs: configuration
> > >>>> format
> > >>>>>>>>> change,
> > >>>>>>>>>>>> improvements in cache APIs, SQL
APIs, transaction mode
> rework.
> > >>>>> The
> > >>>>>>>>>> wishlist
> > >>>>>>>>>>>> of changes for Ignite 3.0 is huge.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> So, I was wondering whether it
makes sense to try to change
> > the
> > >>>>> old
> > >>>>>>>>>>>> codebase, or start with a new baseline
and move old pieces
> of
> > >>>>> code
> > >>>>>>>>> that
> > >>>>>>>>>> do
> > >>>>>>>>>>>> not require significant rework.
Personally, I would go with
> > the
> > >>>>>>> second
> > >>>>>>>>>>>> option for the following reasons:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - We have a chance to shift the
development paradigm in the
> > >>>>> project
> > >>>>>>>>> and
> > >>>>>>>>>>>> introduce the practice of true
unit-tests. In the new
> baseline
> > >>>> at
> > >>>>>> the
> > >>>>>>>>>>>> beginning there will be no ability
to run an end-to-end
> > >>>> scenario,
> > >>>>>>>>> thus
> > >>>>>>>>>> we
> > >>>>>>>>>>>> will be forced to write unit-tests.
So far, such practice
> was
> > >>>>> hard
> > >>>>>> to
> > >>>>>>>>>>>> implement because of tight coupling
between Ignite
> components
> > >>>> and
> > >>>>>>>>>> inability
> > >>>>>>>>>>>> to instantiate components without
an instance of
> > KernalContext.
> > >>>>> For
> > >>>>>>>>>>>> example, we should be able to thoroughly
test internal
> > >>>>> primitives,
> > >>>>>>>>>> such as
> > >>>>>>>>>>>> replication protocol (without actual
communication),
> > >>>> distributed
> > >>>>>>>>>>>> metastorage contracts, persistence
layer, etc.
> > >>>>>>>>>>>> - We will significantly reduce
the development cycle in the
> > >>>>>> beginning
> > >>>>>>>>>>>> (right now the RunAll takes two
hours of astronomical time
> > with
> > >>>>>> empty
> > >>>>>>>>>> TC;
> > >>>>>>>>>>>> in the new approach developer will
be able to run ALL tests
> > >>>>> locally
> > >>>>>>>>> in
> > >>>>>>>>>> a
> > >>>>>>>>>>>> matter of minutes)
> > >>>>>>>>>>>> - We can get rid of TC bot and
enforce green TC by
> integrating
> > >>>> TC
> > >>>>>>>>> build
> > >>>>>>>>>>>> results with GitHub PRs (the same
way Travis is currently
> > >>>>>> integrated
> > >>>>>>>>>> to PR
> > >>>>>>>>>>>> check). We should restrict PR merge
without a TC check
> > >>>>>>>>>>>> - We will still have to re-write
all tests, but only once.
> If
> > >>>> we
> > >>>>>> try
> > >>>>>>>>> to
> > >>>>>>>>>>>> modify the old codebase, we would
need to modify all the
> tests
> > >>>>> for
> > >>>>>>>>>> every
> > >>>>>>>>>>>> major change (public API change,
configuration change)
> > >>>>>>>>>>>> - We will have fewer conflicts
when working together. For
> > >>>>> example,
> > >>>>>> I
> > >>>>>>>>>>>> cannot imagine how one would merge
two changes of getting
> rid
> > >>>> of
> > >>>>>>>>>>>> IgniteFuture and changes in replication
protocol, for
> example
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Technically, I would suggest creating
a new repository for
> > >>>> Ignite
> > >>>>>> 3.0
> > >>>>>>>>>>>> (perhaps, a new clean branch, but
a new repo looks nicer to
> > me)
> > >>>>>> and a
> > >>>>>>>>>> new
> > >>>>>>>>>>>> Ignite 3.0 TeamCity project.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> While it may seem quite radical,
I do believe that this
> > >>>> approach
> > >>>>>> will
> > >>>>>>>>>> give
> > >>>>>>>>>>>> us more benefits than trying to
make such major changes in
> the
> > >>>>>>>>> existing
> > >>>>>>>>>>>> codebase. If needed, let's schedule
a discord chat like
> before
> > >>>> to
> > >>>>>>>>>> discuss
> > >>>>>>>>>>>> this.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>> Andrey V. Mashenkov
> > >>>>
> > >>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message