ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolay Izhikov <nizhi...@apache.org>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Tue, 03 Nov 2020 09:49:17 GMT
Time works for me.

> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <alexey.goncharuk@gmail.com> написал(а):
> 
> Nikolay,
> 
> I am up for the call. I will try to explain my reasoning in greater detail
> and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
> 
> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhikov@apache.org>:
> 
>> Igniters, should we have a call for this topic?
>> 
>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupitsyn@apache.org>
>> написал(а):
>>> 
>>>> not intend to rewrite everything from scratch
>>> 
>>>> Every single test from Ignite 2.x should be moved to Ignite 3
>>>> regardless of how we choose to proceed.
>>> 
>>> Alexey, thank you for the explanation, this addresses all of my concerns.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>> andrey.mashenkov@gmail.com>
>>> wrote:
>>> 
>>>> Hi, Igniters.
>>>> 
>>>> * AFAIU, we need a new repo if we want to apply different restrictions
>> to
>>>> pull requests,
>>>> otherwise I see no difference for myself.
>>>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>>>> checks mandatory.
>>>> 
>>>> I think that relaxed requirements here will lead to bad product quality.
>>>> 
>>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>>>> During active development tests will be broken most of time, so,
>>>> I'd port them e.g. suite-by-suite once we will have a stable and
>> featured
>>>> environment to run them and of course make test's code clear and avoid
>>>> bad/non-relevant ones.
>>>> 
>>>> * I like bottom-up approach.
>>>> With it we could make a better framework. I mean clear component
>> lifecycle,
>>>> component wiring mechanics, general methods to approach core components
>>>> such as exchange/communication
>>>> to avoid code mess like we have in ExchangeFuture with all these custom
>>>> callbacks for each component, interfaces like
>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>>>> a pack of
>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>>>> and so on in various unexpected places.
>>>> Hope, we will be able to port most of the good code to the new framework
>>>> version.
>>>> 
>>>> 
>>>> 
>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>>>> alexey.goncharuk@gmail.com>
>>>> wrote:
>>>> 
>>>>> Nikolay, Pavel,
>>>>> 
>>>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>>>> intend to rewrite everything from scratch (I never used this phrase).
>>>> There
>>>>> are significant parts of code that would be moved with minimal
>>>>> modifications. Second, I never said that we will get rid of the old
>> tests
>>>>> codebase. Every single test from Ignite 2.x should be moved to Ignite
3
>>>>> regardless of how we choose to proceed.
>>>>> 
>>>>> My point is that for some parts of the code a clean bottom-up
>>>>> implementation will be cheaper in many ways. Let me give you a few
>>>> concrete
>>>>> examples:
>>>>> 
>>>>>  - I think no one can object that we need a cleanly separated
>>>> persistence
>>>>>  layer for Ignite. There is a very raw draft IEP for this already. On
>>>> the
>>>>>  other hand, I think we also can agree that we need a split-brain
>>>>> resistant
>>>>>  replication protocol for caches. There is also an IEP for this.
>>>> Neither
>>>>> of
>>>>>  the changes is a good fit for 2.x because they are likely to
>> introduce
>>>>>  breaking changes in the persistence layer, configuration and
>> behavior.
>>>>>  Additionally, these components are now tightly coupled, so there is
>> no
>>>>> way
>>>>>  these two changes can be implemented in parallel and then merged
>>>>> together
>>>>>  easily. So what we will end up with is having to implement these
>>>> changes
>>>>>  sequentially, fixing all existing tests twice, and essentially
>>>> throwing
>>>>>  away half of the work done because the other part of the change is
>>>>>  re-implemented
>>>>>  - Similar example goes with getting rid of IgniteInternalFuture and
>>>>>  replacing it with CompletableFuture, and any other change that
>> touches
>>>>> the
>>>>>  asynchronous part of the code.
>>>>> 
>>>>> Third, I do not see how this choice affects the UX of Ignite. The end
>>>> user
>>>>> experience must be fixed in the IEP regardless of the development
>> process
>>>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>>>> that.
>>>>> 
>>>>> Pavel, agree that a repo/branch is a technicality, I guess if
>>>> reformulate,
>>>>> my point is that we might agree to have a single development master
>>>> branch
>>>>> with 'disassembled' end-to-end functionality for some period of time
to
>>>>> speed up development, and re-assemble the core features after having
>>>>> submodules tested independently.
>>>>> 
>>>>> Nikolay,
>>>>>> We have many features that have to evolve.
>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> This is not very specific. In the end, resources are limited and we
>> will
>>>>> not be able to drive both tracks simultaneously, especially after a
>>>> couple
>>>>> of features having been implemented for Ignite 3.0. If there are indeed
>>>>> some major changes that we want to do in Ignite 2.x instead of putting
>>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
>> that's
>>>>> why I am eager to move forward with Ignite 3.0.
>>>>> 
>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>> These changes will be covered by end-to-end tests and migrated to
>> Ignite
>>>>> 3.0, so I see no issues here.
>>>>> 
>>>>> Finally, Anton & Nikolay
>>>>> I do not have an estimate for this simply because the activity is
>>>>> community-driven and it depends on the number of people willing to
>>>>> contribute. With the current pace, I would hope to have an RC of Ignite
>>>> 3.0
>>>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>>>> incremental changes, we will not be able to implement even half of the
>>>>> wishlist by that time.
>>>>> I doubt that releasing several major releases with breaking changes
>> will
>>>>> make Ignite users happy either because each upgrade will cost Ignite
>>>> users
>>>>> money, so the fewer major versions we release, the better. Thus my wish
>>>> to
>>>>> include all breaking changes in one release.
>>>>> 
>>>>> I'll be now quiet for a while, let's see what other community members
>>>>> think.
>>>>> 
>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupitsyn@apache.org>:
>>>>> 
>>>>>> 1. Rewriting from scratch is never a good idea.
>>>>>> We don't want to follow the path of Netscape and lose all our users
>>>>>> by the time we have a working 3.0 [1]
>>>>>> 
>>>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>>>> the
>>>>>> problem with a branch?
>>>>>> 
>>>>>> 3. We should keep existing integration tests when possible.
>>>>>> We have accumulated a lot of edge case knowledge over the years,
>>>>>> it is not a good idea to send all of that down the drain.
>>>>>> Yes, integration tests are slow, but they are the most valuable.
>>>>>> I think we can move more stuff into nightly runs and have a fast
and
>>>>> modern
>>>>>> basic suite.
>>>>>> 
>>>>>> 
>>>>>> Alexey, you are much more familiar with the Ignite core codebase
than
>>>>> most
>>>>>> of us,
>>>>>> can you please explain in more detail which particular feature, in
>> your
>>>>>> opinion,
>>>>>> mandates this "start from scratch" approach?
>>>>>> Is it really not possible at all to follow a less radical way?
>>>>>> 
>>>>>> 
>>>>>> [1]
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>>> 
>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhikov@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hello, Alexey.
>>>>>>> 
>>>>>>> I think that «rewriting from scratch» approach has a high risk
to
>>>> make
>>>>>> new
>>>>>>> features unusable.
>>>>>>> At the time Ignite2 was started no-one wants to do bad UX or
bad
>>>>>> features.
>>>>>>> Nevertheless, it happen.
>>>>>>> 
>>>>>>> I think we can avoid it with the Ignite3 and successors if we
will
>>>> move
>>>>>>> step by step without keeping backward compatibility
>>>>>>> With the step by step approach, we can focus on each component
>>>>>> separately.
>>>>>>> 
>>>>>>>> What new features are we planning to implement for Ignite
2.x?
>>>>>>> 
>>>>>>> We have many features that have to evolve.
>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>>>> backward
>>>>>>> compatibility.
>>>>>>> We have many users who are happy with the 2.x with all it’s
issues.
>>>>>>> 
>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <av@apache.org>
>>>>> написал(а):
>>>>>>>> 
>>>>>>>> Alexey,
>>>>>>>> 
>>>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>>>> production-ready
>>>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>>> 
>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Nikolay,
>>>>>>>>> 
>>>>>>>>> What new features are we planning to implement for Ignite
2.x? I
>>>>> think
>>>>>>> once
>>>>>>>>> we commence working on Ignite 3.0, we should gradually
cease the
>>>>>>> activity
>>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel
development
>>>>> will
>>>>>> be
>>>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>>>> 
>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov
<nizhikov@apache.org
>>>>> :
>>>>>>>>> 
>>>>>>>>>> To be clear:
>>>>>>>>>> 
>>>>>>>>>>> I would suggest creating a new repository for
Ignite 3.0
>>>>> (perhaps, a
>>>>>>>>> new
>>>>>>>>>> clean branch, but a new repo looks nicer to me) and
a new Ignite
>>>>> 3.0
>>>>>>>>>> TeamCity project.
>>>>>>>>>> 
>>>>>>>>>> +1 for new Team City project.
>>>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>>>> -1 for new repo.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov
<
>>>> NIzhikov.dev@gmail.com
>>>>>> 
>>>>>>>>>> написал(а):
>>>>>>>>>>> 
>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>> 
>>>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>>>> Developing new features for 2 separate branches
with the
>>>> different
>>>>>>> APIs
>>>>>>>>>> and internal structure is overwhelming
>>>>>>>>>>> 
>>>>>>>>>>> Maybe we should relax a bit requirements for
Ignite3?
>>>>>>>>>>> Maybe we should move step by step and make Ignite3
with new
>>>>>>>>>> configuration than Ignite4 with new transactions,
etc?
>>>>>>>>>>> 
>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk
<
>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>> написал(а):
>>>>>>>>>>>> 
>>>>>>>>>>>> Igniters,
>>>>>>>>>>>> 
>>>>>>>>>>>> I wanted to pitch a rather radical idea regarding
the Ignite
>>>> 3.0
>>>>>>>>>>>> development which has occurred to me some
time ago.
>>>>>>>>>>>> 
>>>>>>>>>>>> We already have several IEPs targeted to
Ignite 3.0 which imply
>>>>>> major
>>>>>>>>>>>> changes to the codebase (the change in replication
protocol and
>>>>>> thus
>>>>>>>>>>>> transactions, change in binary format, updated
metastorage,
>>>> etc).
>>>>>> We
>>>>>>>>>> also
>>>>>>>>>>>> planned significant changes in public APIs:
configuration
>>>> format
>>>>>>>>> change,
>>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction
mode rework.
>>>>> The
>>>>>>>>>> wishlist
>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>>> 
>>>>>>>>>>>> So, I was wondering whether it makes sense
to try to change the
>>>>> old
>>>>>>>>>>>> codebase, or start with a new baseline and
move old pieces of
>>>>> code
>>>>>>>>> that
>>>>>>>>>> do
>>>>>>>>>>>> not require significant rework. Personally,
I would go with the
>>>>>>> second
>>>>>>>>>>>> option for the following reasons:
>>>>>>>>>>>> 
>>>>>>>>>>>> - We have a chance to shift the development
paradigm in the
>>>>> project
>>>>>>>>> and
>>>>>>>>>>>> introduce the practice of true unit-tests.
In the new baseline
>>>> at
>>>>>> the
>>>>>>>>>>>> beginning there will be no ability to run
an end-to-end
>>>> scenario,
>>>>>>>>> thus
>>>>>>>>>> we
>>>>>>>>>>>> will be forced to write unit-tests. So far,
such practice was
>>>>> hard
>>>>>> to
>>>>>>>>>>>> implement because of tight coupling between
Ignite components
>>>> and
>>>>>>>>>> inability
>>>>>>>>>>>> to instantiate components without an instance
of KernalContext.
>>>>> For
>>>>>>>>>>>> example, we should be able to thoroughly
test internal
>>>>> primitives,
>>>>>>>>>> such as
>>>>>>>>>>>> replication protocol (without actual communication),
>>>> distributed
>>>>>>>>>>>> metastorage contracts, persistence layer,
etc.
>>>>>>>>>>>> - We will significantly reduce the development
cycle in the
>>>>>> beginning
>>>>>>>>>>>> (right now the RunAll takes two hours of
astronomical time with
>>>>>> empty
>>>>>>>>>> TC;
>>>>>>>>>>>> in the new approach developer will be able
to run ALL tests
>>>>> locally
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>> matter of minutes)
>>>>>>>>>>>> - We can get rid of TC bot and enforce green
TC by integrating
>>>> TC
>>>>>>>>> build
>>>>>>>>>>>> results with GitHub PRs (the same way Travis
is currently
>>>>>> integrated
>>>>>>>>>> to PR
>>>>>>>>>>>> check). We should restrict PR merge without
a TC check
>>>>>>>>>>>> - We will still have to re-write all tests,
but only once. If
>>>> we
>>>>>> try
>>>>>>>>> to
>>>>>>>>>>>> modify the old codebase, we would need to
modify all the tests
>>>>> for
>>>>>>>>>> every
>>>>>>>>>>>> major change (public API change, configuration
change)
>>>>>>>>>>>> - We will have fewer conflicts when working
together. For
>>>>> example,
>>>>>> I
>>>>>>>>>>>> cannot imagine how one would merge two changes
of getting rid
>>>> of
>>>>>>>>>>>> IgniteFuture and changes in replication protocol,
for example
>>>>>>>>>>>> 
>>>>>>>>>>>> Technically, I would suggest creating a new
repository for
>>>> Ignite
>>>>>> 3.0
>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo
looks nicer to me)
>>>>>> and a
>>>>>>>>>> new
>>>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>>> 
>>>>>>>>>>>> While it may seem quite radical, I do believe
that this
>>>> approach
>>>>>> will
>>>>>>>>>> give
>>>>>>>>>>>> us more benefits than trying to make such
major changes in the
>>>>>>>>> existing
>>>>>>>>>>>> codebase. If needed, let's schedule a discord
chat like before
>>>> to
>>>>>>>>>> discuss
>>>>>>>>>>>> this.
>>>>>>>>>>>> 
>>>>>>>>>>>> WDYT?
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>> 
>> 
>> 


Mime
View raw message