sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Attila Szabó <mau...@apache.org>
Subject Re: Sqoop CI changes
Date Fri, 07 Apr 2017 11:04:41 GMT
Hello everyone,

First of all I'd like to thank that Anna is willing to invest some efforts
on making our CI system better, and sorry for my delayed answer ( however I
do still hope it helps regardless the stage of the ongoing efforts )

I'd like to share the following thoughts here:
- Although it would make sense to eliminate the four ( right now totally
equal ) CI cycles and create only one, but regardless some "static noise"
it doesn't cause any serious issues for our current commit flow.
- Creating a precommit hook sounds like a great idea, I would encourage the
community to move on that path.
- However: According to my humblest opinion the biggest problem with the
current CI is that it doesn't execute the so called " 3rd party tests" (
which is generally our DB integration test layer), and thus it provides
only a limited safety belt for us ( and we've seen quite a few regression
on this front in the past one year ). Although we do have solution for
running those tests manually from command line, it's quite difficult to
setup/test those things from a single desktop, thus cause serious
difficulties in validating some changeset before commit.
- Still an issue we have on this front is our build
system/scripts/mechanism, which again could slow down the Dev+commit flow.
Although we've started efforts on this front, the final solution was not
fully delivered yet.

As a conclusion of the elements above:
Anna! Would you mind first focusing on the build scripts and the "3rd
party" CI automation instead of eliminating the obsolete stuff? IMHO that
would be a much better usage of your efforts and would provide a much
bigger impact for the community.

With my kindest regards,
Attila

On Mar 28, 2017 2:44 PM, "Erzsebet Szilagyi" <liz.szilagyi@cloudera.com>
wrote:

> Great ideas!
>
> I agree with Bogi and Szabolcs on the redundant test jobs.
>
> Would this pre-commit hook launch the same process as the current
> post-commit hook, or would this do something different?
> I think in the first case we could rework the post-commit check into the
> pre-commit hook, in the latter I'm curious about what exactly this check
> would add.
> In general I support the idea: we have seen a number of problems that could
> have been avoided, so this shall be a very useful change!
>
> Thank you,
> Liz
>
> On Fri, Mar 24, 2017 at 9:39 AM, Szabolcs Vasas <vasas@cloudera.com>
> wrote:
>
> > Hi Anna,
> >
> > Removing the redundant test execution jobs sounds great, I think you can
> go
> > ahead with that.
> >
> > Regarding the pre-commit hook: what would be the purpose of it exactly?
> > Would it execute the unit tests before the patch is committed?
> >
> > Regards,
> > Szabolcs
> >
> > On Thu, Mar 23, 2017 at 4:03 PM, Anna Szonyi <szonyi@cloudera.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I would like to make the following changes to the Sqoop CI system:
> > > Disable the SCM polling for the Sqoop-hadoop23 Sqoop-hadoop20 and
> > > Sqoop-hadoop100 jobs (and later delete the jobs themselves),
> > > as the current trunk version of sqoop no longer contains these
> profiles,
> > so
> > > these runs are redundant.
> > >
> > > I would also like to propose the creation of a pre-commit hook for
> Sqoop
> > > (like the existing one for Sqoop2).
> > >
> > > Please let me know if you have any objections.
> > >
> > > Thanks,
> > > Anna
> > >
> >
> >
> >
> > --
> > Szabolcs Vasas
> > Software Engineer
> > <http://www.cloudera.com>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message