ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Petr Ivanov <mr.wei...@gmail.com>
Subject Re: [DISCUSS] Ignite 3.0 development approach
Date Fri, 27 Nov 2020 13:48:34 GMT
> Storing CI/CD code (yaml definitions for Travis/Azure/GH Actions, Jenkins
> pipelines, etc) in the same repo is very common.
> Secrets are stored separately (e.g. GitHub secrets), TeamCity probably has
> a similar feature.


My main security concern — anyone, including third-party person without even committer permissions
can modify build configuration in such a way that it will be doing something not intend to
do (bitcoin mining for instance) or even something harmful (like trying to attack underlying
TC infrastructure). If we are to store CI/CD configuration alongside code, some restrictions
are required.



> On 27 Nov 2020, at 15:59, Pavel Tupitsyn <ptupitsyn@apache.org> wrote:
> 
>> migrating to 'CI/CD as a Code'
> 
> Huge +1 for this.
> 
> 
>> where should the code be stored ..
>> alongside project's code (can be possible security hole)
> 
> Storing CI/CD code (yaml definitions for Travis/Azure/GH Actions, Jenkins
> pipelines, etc) in the same repo is very common.
> Secrets are stored separately (e.g. GitHub secrets), TeamCity probably has
> a similar feature.
> 
> 
> On Fri, Nov 27, 2020 at 3:28 PM Petr Ivanov <mr.weider@gmail.com> wrote:
> 
>> More or less, unless we specifically forbid that, I guess
>> 
>> However there is bigger concern: JDK 15 is STS, so after half of a year it
>> will be out of support and no major production team will use that JDK in
>> their environment.
>> I would stick to JDK 11 as it is LTS at least until JDK17, plus — a lot of
>> efforts should be made to enforce Apache Ignite be built on JDK 11 alone,
>> not to mention 15th version...
>> 
>> 
>> 
>> Also, If we are going to introduce such major changes, I'd like to purpose
>> maven project refactoring as well:
>> 1. Full revision of all ant-calling tasks with javascript functions and
>> alike — the complexity of those are overwhelming, something new should be
>> at least researched.
>> 2. Full revisions of profiles (for both root and modules) as there are
>> some obsolete ones, and some that do ambiguous or, even worse, totally
>> unknown tasks.
>> 3. Introduce plugin and dependency management sections to control over and
>> concrete versions of software we are relying in our project. Additionally —
>> add BOM with all Ignite modules and their dependencies, which should help
>> our users to better embed Ignite to their projects.
>> 4. Up all versions of plugins and dependencies where possible to there
>> current production versions (for plugins — it should be a must if we are
>> ever going to build project under latest JDK versions).
>> 5. Prepare project for parallel building, testing and assembling where
>> possible for faster development process, with possible additional
>> enhancement like incremental rebuild.
>> 6. Possibly — research alternate builders, like Gradle (thought there are
>> a lot of questions to its race condition issues during multimodule builds).
>> 
>> 
>> 
>> And last, but not least — think of migrating to 'CI/CD as a Code' approach
>> for our main instrument TeamCity.
>> Whole project (both test and release build configurations) can be
>> described using DSL (Kotlin in case of TC) and stored in VCS, forcing
>> infrastructure changes to go through the same development processes
>> including discussions, review, change history and etc.
>> 
>> Only I am not sure for now about where should the code be stored — in
>> separate repository (secure, but disables testing of code with TC settings
>> both in single PR), or alongside project's code (can be possible security
>> hole).
>> That would require additional dev thread I think.
>> 
>> 
>> 
>> WDYT?
>> 
>>> On 24 Nov 2020, at 20:04, Pavel Tupitsyn <ptupitsyn@apache.org> wrote:
>>> 
>>> If we use Java15 for development, can the resulting package be used from
>> a
>>> Java11 app (the latest LTS)?
>>> 
>>> On Tue, Nov 24, 2020 at 7:51 PM Andrey Mashenkov <
>> andrey.mashenkov@gmail.com>
>>> wrote:
>>> 
>>>> Jave15 looks awesome.
>>>> 
>>>> * Hidden classes [1] can be used by codegenerators.
>>>> * Records [2] can replace boilerplate code like IgniteBiTuple,
>> GridTupleX.
>>>> 
>>>> [1] https://openjdk.java.net/jeps/371
>>>> [2] https://openjdk.java.net/jeps/384
>>>> 
>>>> On Tue, Nov 24, 2020 at 3:38 PM Alexey Zinoviev <zaleslaw.sin@gmail.com
>>> 
>>>> wrote:
>>>> 
>>>>> Java 15 is better, VarHandles, ForeignMemory access and so on.
>>>>> 
>>>>> In both cases, I support the Java version 11 and higher for the
>>>>> development!
>>>>> 
>>>>> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
>>>> andrey.mashenkov@gmail.com
>>>>>> :
>>>>> 
>>>>>> Let's add maven plugins  for sanity checks at the early stage.
>>>>>> I've created a ticket for this [1].
>>>>>> 
>>>>>> Also, I've found initial pom.xml has a target version Java 8.
>>>>>> Do we intend to move to Java 11 (or may be next LTS) and drop Java
8
>> in
>>>>>> Ignite 3.0?
>>>>>> 
>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13751
>>>>>> 
>>>>>> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>> 
>>>>>>> Folks,
>>>>>>> 
>>>>>>> I went ahead and created the repository [1]. I also configured
a
>>>>> TeamCity
>>>>>>> project [2] that runs all available JUnit tests on every PR creation
>>>> or
>>>>>>> update. It also sends the status update to GitHub so that it's
>>>>> reflected
>>>>>> in
>>>>>>> the PR itself so that we can do merges directly from GitHub.
Basic
>>>>> steps
>>>>>> to
>>>>>>> make a change are described on the Wiki page [3].
>>>>>>> 
>>>>>>> Let me know if you have any questions.
>>>>>>> 
>>>>>>> [1] https://github.com/apache/ignite-3
>>>>>>> [2] https://ci.ignite.apache.org/project/ignite3
>>>>>>> [3]
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess
>>>>>>> 
>>>>>>> -Val
>>>>>>> 
>>>>>>> On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
>>>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Thanks, guys. It looks like we are much closer to the consensus
>>>> now.
>>>>> I
>>>>>>>> totally on board with the plan, but I would also like to
address
>>>> the
>>>>>>>> short-term needs. As I've already mentioned earlier, there
are
>>>>> several
>>>>>>>> active IEPs, but we still don't have even a preliminary technical
>>>>>> process
>>>>>>>> for working on these IEPs. I believe this might be frustrating
for
>>>>> the
>>>>>>>> folks who would like to commit code.
>>>>>>>> 
>>>>>>>> The scope we agreed on is quite big, and it will surely take
>>>>>> significant
>>>>>>>> time to implement all the changes and stabilize them. Therefore,
>>>> it's
>>>>>>> clear
>>>>>>>> to me that we will have to maintain 2.x and 3.x in parallel
for
>>>> quite
>>>>>>> some
>>>>>>>> time - this needs to be addressed somehow. I'm convinced
that
>>>> having
>>>>> a
>>>>>>>> separate repo is the ONLY way to do that, and so far, I haven't
>>>> heard
>>>>>> any
>>>>>>>> clear alternatives or reasons why we shouldn't do this.
>>>>>>>> 
>>>>>>>> That said, I'm inclined to proceed with this in the next
few days
>>>> - I
>>>>>>> will
>>>>>>>> create a repo and describe the process (which we, of course,
can
>>>>>> discuss
>>>>>>>> and modify going forward). Let's, at the very least, try
and see
>>>>> where
>>>>>> it
>>>>>>>> leads us.
>>>>>>>> 
>>>>>>>> If someone has any concrete alternative options on how to
we can
>>>>>> maintain
>>>>>>>> two major versions in parallel, let's have another voice
discussion
>>>>>> this
>>>>>>>> Friday. If we do the meeting, we should set it up with a
clear goal
>>>>> to
>>>>>>> make
>>>>>>>> a decision. Please let me know if there is interest in this.
>>>>>>>> 
>>>>>>>> -Val
>>>>>>>> 
>>>>>>>> On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk <
>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Good,
>>>>>>>>> 
>>>>>>>>> I think we have an intermediate agreement on the scope
and
>>>>>> significance
>>>>>>> of
>>>>>>>>> the changes we want to make. I suggest creating separate
>>>> discussion
>>>>>>>>> streams
>>>>>>>>> and calls for each of the suggested topics so that:
>>>>>>>>> 
>>>>>>>>>  - It is clear for the community what is the motivation
of the
>>>>>> stream
>>>>>>>>>  (this includes both functional targets and technical
debt
>>>> issues
>>>>>>>>> pointed
>>>>>>>>>  out by Sergey)
>>>>>>>>>  - Who is planning to take an active part in each of
the streams
>>>>>> (i.e.
>>>>>>>>>  the 'design committee', as Sergey suggested)
>>>>>>>>>  - What are the intermediate and final goals for each
of the
>>>>> streams
>>>>>>>>>  - What are the cross-stream interactions and how we
integrate
>>>>> them
>>>>>>>>>  - How each of the streams will be integrated with the
current
>>>>>>> codebase
>>>>>>>>>  based on the above (here is where we will see whether
drop-in
>>>> or
>>>>>>>>>  incremental approaches make more sense)
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Best regards,
>>>>>> Andrey V. Mashenkov
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>> 
>> 
>> 


Mime
View raw message