ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolay Izhikov <nizhi...@apache.org>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Tue, 03 Nov 2020 07:09:21 GMT
Igniters, should we have a call for this topic?

> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org> написал(а):
> 
>> not intend to rewrite everything from scratch
> 
>> Every single test from Ignite 2.x should be moved to Ignite 3
>> regardless of how we choose to proceed.
> 
> Alexey, thank you for the explanation, this addresses all of my concerns.
> 
> 
> 
> 
> 
> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <andrey.mashenkov@gmail.com>
> wrote:
> 
>> Hi, Igniters.
>> 
>> * AFAIU, we need a new repo if we want to apply different restrictions to
>> pull requests,
>> otherwise I see no difference for myself.
>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>> checks mandatory.
>> 
>> I think that relaxed requirements here will lead to bad product quality.
>> 
>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>> During active development tests will be broken most of time, so,
>> I'd port them e.g. suite-by-suite once we will have a stable and featured
>> environment to run them and of course make test's code clear and avoid
>> bad/non-relevant ones.
>> 
>> * I like bottom-up approach.
>> With it we could make a better framework. I mean clear component lifecycle,
>> component wiring mechanics, general methods to approach core components
>> such as exchange/communication
>> to avoid code mess like we have in ExchangeFuture with all these custom
>> callbacks for each component, interfaces like
>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>> a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>> and so on in various unexpected places.
>> Hope, we will be able to port most of the good code to the new framework
>> version.
>> 
>> 
>> 
>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>> alexey.goncharuk@gmail.com>
>> wrote:
>> 
>>> Nikolay, Pavel,
>>> 
>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>> intend to rewrite everything from scratch (I never used this phrase).
>> There
>>> are significant parts of code that would be moved with minimal
>>> modifications. Second, I never said that we will get rid of the old tests
>>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
>>> regardless of how we choose to proceed.
>>> 
>>> My point is that for some parts of the code a clean bottom-up
>>> implementation will be cheaper in many ways. Let me give you a few
>> concrete
>>> examples:
>>> 
>>>   - I think no one can object that we need a cleanly separated
>> persistence
>>>   layer for Ignite. There is a very raw draft IEP for this already. On
>> the
>>>   other hand, I think we also can agree that we need a split-brain
>>> resistant
>>>   replication protocol for caches. There is also an IEP for this.
>> Neither
>>> of
>>>   the changes is a good fit for 2.x because they are likely to introduce
>>>   breaking changes in the persistence layer, configuration and behavior.
>>>   Additionally, these components are now tightly coupled, so there is no
>>> way
>>>   these two changes can be implemented in parallel and then merged
>>> together
>>>   easily. So what we will end up with is having to implement these
>> changes
>>>   sequentially, fixing all existing tests twice, and essentially
>> throwing
>>>   away half of the work done because the other part of the change is
>>>   re-implemented
>>>   - Similar example goes with getting rid of IgniteInternalFuture and
>>>   replacing it with CompletableFuture, and any other change that touches
>>> the
>>>   asynchronous part of the code.
>>> 
>>> Third, I do not see how this choice affects the UX of Ignite. The end
>> user
>>> experience must be fixed in the IEP regardless of the development process
>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>> that.
>>> 
>>> Pavel, agree that a repo/branch is a technicality, I guess if
>> reformulate,
>>> my point is that we might agree to have a single development master
>> branch
>>> with 'disassembled' end-to-end functionality for some period of time to
>>> speed up development, and re-assemble the core features after having
>>> submodules tested independently.
>>> 
>>> Nikolay,
>>>> We have many features that have to evolve.
>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>> This is not very specific. In the end, resources are limited and we will
>>> not be able to drive both tracks simultaneously, especially after a
>> couple
>>> of features having been implemented for Ignite 3.0. If there are indeed
>>> some major changes that we want to do in Ignite 2.x instead of putting
>>> effort into 3.0 - let's discuss them. I am just not aware of any, that's
>>> why I am eager to move forward with Ignite 3.0.
>>> 
>>>> We have bugs and issues that can be fixed in 2.x without breaking
>> backward
>>> compatibility.
>>>> We have many users who are happy with the 2.x with all it’s issues.
>>> These changes will be covered by end-to-end tests and migrated to Ignite
>>> 3.0, so I see no issues here.
>>> 
>>> Finally, Anton & Nikolay
>>> I do not have an estimate for this simply because the activity is
>>> community-driven and it depends on the number of people willing to
>>> contribute. With the current pace, I would hope to have an RC of Ignite
>> 3.0
>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>> incremental changes, we will not be able to implement even half of the
>>> wishlist by that time.
>>> I doubt that releasing several major releases with breaking changes will
>>> make Ignite users happy either because each upgrade will cost Ignite
>> users
>>> money, so the fewer major versions we release, the better. Thus my wish
>> to
>>> include all breaking changes in one release.
>>> 
>>> I'll be now quiet for a while, let's see what other community members
>>> think.
>>> 
>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org>:
>>> 
>>>> 1. Rewriting from scratch is never a good idea.
>>>> We don't want to follow the path of Netscape and lose all our users
>>>> by the time we have a working 3.0 [1]
>>>> 
>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>> the
>>>> problem with a branch?
>>>> 
>>>> 3. We should keep existing integration tests when possible.
>>>> We have accumulated a lot of edge case knowledge over the years,
>>>> it is not a good idea to send all of that down the drain.
>>>> Yes, integration tests are slow, but they are the most valuable.
>>>> I think we can move more stuff into nightly runs and have a fast and
>>> modern
>>>> basic suite.
>>>> 
>>>> 
>>>> Alexey, you are much more familiar with the Ignite core codebase than
>>> most
>>>> of us,
>>>> can you please explain in more detail which particular feature, in your
>>>> opinion,
>>>> mandates this "start from scratch" approach?
>>>> Is it really not possible at all to follow a less radical way?
>>>> 
>>>> 
>>>> [1]
>>>> 
>>>> 
>>> 
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>> 
>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org>
>>>> wrote:
>>>> 
>>>>> Hello, Alexey.
>>>>> 
>>>>> I think that «rewriting from scratch» approach has a high risk to
>> make
>>>> new
>>>>> features unusable.
>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
>>>> features.
>>>>> Nevertheless, it happen.
>>>>> 
>>>>> I think we can avoid it with the Ignite3 and successors if we will
>> move
>>>>> step by step without keeping backward compatibility
>>>>> With the step by step approach, we can focus on each component
>>>> separately.
>>>>> 
>>>>>> What new features are we planning to implement for Ignite 2.x?
>>>>> 
>>>>> We have many features that have to evolve.
>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>> 
>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org>
>>> написал(а):
>>>>>> 
>>>>>> Alexey,
>>>>>> 
>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>> production-ready
>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>> 
>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>> alexey.goncharuk@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Nikolay,
>>>>>>> 
>>>>>>> What new features are we planning to implement for Ignite 2.x?
I
>>> think
>>>>> once
>>>>>>> we commence working on Ignite 3.0, we should gradually cease
the
>>>>> activity
>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
>>> will
>>>> be
>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>> 
>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhikov@apache.org
>>> :
>>>>>>> 
>>>>>>>> To be clear:
>>>>>>>> 
>>>>>>>>> I would suggest creating a new repository for Ignite
3.0
>>> (perhaps, a
>>>>>>> new
>>>>>>>> clean branch, but a new repo looks nicer to me) and a new
Ignite
>>> 3.0
>>>>>>>> TeamCity project.
>>>>>>>> 
>>>>>>>> +1 for new Team City project.
>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>> -1 for new repo.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>> NIzhikov.dev@gmail.com
>>>> 
>>>>>>>> написал(а):
>>>>>>>>> 
>>>>>>>>> Hello, Alexey.
>>>>>>>>> 
>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>> Developing new features for 2 separate branches with
the
>> different
>>>>> APIs
>>>>>>>> and internal structure is overwhelming
>>>>>>>>> 
>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
>>>>>>>>> Maybe we should move step by step and make Ignite3 with
new
>>>>>>>> configuration than Ignite4 with new transactions, etc?
>>>>>>>>> 
>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk
<
>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>> написал(а):
>>>>>>>>>> 
>>>>>>>>>> Igniters,
>>>>>>>>>> 
>>>>>>>>>> I wanted to pitch a rather radical idea regarding
the Ignite
>> 3.0
>>>>>>>>>> development which has occurred to me some time ago.
>>>>>>>>>> 
>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0
which imply
>>>> major
>>>>>>>>>> changes to the codebase (the change in replication
protocol and
>>>> thus
>>>>>>>>>> transactions, change in binary format, updated metastorage,
>> etc).
>>>> We
>>>>>>>> also
>>>>>>>>>> planned significant changes in public APIs: configuration
>> format
>>>>>>> change,
>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction
mode rework.
>>> The
>>>>>>>> wishlist
>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>> 
>>>>>>>>>> So, I was wondering whether it makes sense to try
to change the
>>> old
>>>>>>>>>> codebase, or start with a new baseline and move old
pieces of
>>> code
>>>>>>> that
>>>>>>>> do
>>>>>>>>>> not require significant rework. Personally, I would
go with the
>>>>> second
>>>>>>>>>> option for the following reasons:
>>>>>>>>>> 
>>>>>>>>>> - We have a chance to shift the development paradigm
in the
>>> project
>>>>>>> and
>>>>>>>>>> introduce the practice of true unit-tests. In the
new baseline
>> at
>>>> the
>>>>>>>>>> beginning there will be no ability to run an end-to-end
>> scenario,
>>>>>>> thus
>>>>>>>> we
>>>>>>>>>> will be forced to write unit-tests. So far, such
practice was
>>> hard
>>>> to
>>>>>>>>>> implement because of tight coupling between Ignite
components
>> and
>>>>>>>> inability
>>>>>>>>>> to instantiate components without an instance of
KernalContext.
>>> For
>>>>>>>>>> example, we should be able to thoroughly test internal
>>> primitives,
>>>>>>>> such as
>>>>>>>>>> replication protocol (without actual communication),
>> distributed
>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>> - We will significantly reduce the development cycle
in the
>>>> beginning
>>>>>>>>>> (right now the RunAll takes two hours of astronomical
time with
>>>> empty
>>>>>>>> TC;
>>>>>>>>>> in the new approach developer will be able to run
ALL tests
>>> locally
>>>>>>> in
>>>>>>>> a
>>>>>>>>>> matter of minutes)
>>>>>>>>>> - We can get rid of TC bot and enforce green TC by
integrating
>> TC
>>>>>>> build
>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
>>>> integrated
>>>>>>>> to PR
>>>>>>>>>> check). We should restrict PR merge without a TC
check
>>>>>>>>>> - We will still have to re-write all tests, but only
once. If
>> we
>>>> try
>>>>>>> to
>>>>>>>>>> modify the old codebase, we would need to modify
all the tests
>>> for
>>>>>>>> every
>>>>>>>>>> major change (public API change, configuration change)
>>>>>>>>>> - We will have fewer conflicts when working together.
For
>>> example,
>>>> I
>>>>>>>>>> cannot imagine how one would merge two changes of
getting rid
>> of
>>>>>>>>>> IgniteFuture and changes in replication protocol,
for example
>>>>>>>>>> 
>>>>>>>>>> Technically, I would suggest creating a new repository
for
>> Ignite
>>>> 3.0
>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks
nicer to me)
>>>> and a
>>>>>>>> new
>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>> 
>>>>>>>>>> While it may seem quite radical, I do believe that
this
>> approach
>>>> will
>>>>>>>> give
>>>>>>>>>> us more benefits than trying to make such major changes
in the
>>>>>>> existing
>>>>>>>>>> codebase. If needed, let's schedule a discord chat
like before
>> to
>>>>>>>> discuss
>>>>>>>>>> this.
>>>>>>>>>> 
>>>>>>>>>> WDYT?
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> --
>> Best regards,
>> Andrey V. Mashenkov
>> 


Mime
View raw message