sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Attila Szabo <mau...@apache.org>
Subject Release 1.5 (maybe 1.4.7 ? ), CI, build system, etc.
Date Wed, 30 Nov 2016 20:27:18 GMT
Dear PMC, dear community,

In the past few months some of us has been already identified that there
would be a need to improve the test coverage, cleanup the CI system, make
the build system much more straightforward for the trunk version of Sqoop.

I think our goals are very simple here:

   - It's very difficult to onboard new committers for the community if the
   component itself is very difficult to test/learn to test, and the CI/JIRA
   system sends out false failures after each commit.
   - It would be also nice to leverage again from the safety belt of CI,
   and not push all of the testing burden on the shoulder of the committer
   alone (of course the contributors has to do tests, but as said it's not
   straightforward now to execute all tests, and also it's still the
   committers responsibility to ensure nothing is broken after a commit, thus
   every safety belt there is needed).


It's been also identified that it would be good to release a new version of
trunk, thus including the improvements of the past 8-9 months, and also
show the livingness of the component for the outside of the world.

Most probably because of lack of time we didn't achieve all of these goals
in the past few months. Thus right now I'd like to take the initiative, and
share my thoughts and goals with you, trying to push forward the above
mentioned things, but of course before I'd like to collect the invaluable
input and wisdom of the community.

The plan would be the following:

   - A few weeks ago I've opened three JIRAs (depending on each other
   linearly) upstream (SQOOP-3050, SQOOP-3051 and SQOOP-3052).
   - SQOOP-3050 is about fix the current test cases (mainly failing because
   of configuration issues)
   - SQOOP-3051 is about cleaning up the build.xml from the obsolete
   profiles (e.g. currently 20,23,100 are failing with tons of test cases, and
   even 200 is not able to correctly run HCatalog tests for example because of
   incompatible class changes).
   - SQOOP-3052 is about to create a new and more robust build system for
   Sqoop (e.g. Maven/Gradle, maybe both)
   - Anna Szonyi (a quite new contributor, but quite active in the past 1
   month) has jumped on SQOOP-3050 and it seems she's finished with those
   efforts (so all unit+third_party test cases are running with the recently
   created hadoop260 profile).
   - AFAIK Anna is also willing to jump on SQOOP-3051, which would be about
   cleanup the old profiles from the build.xml and ivysettings, and keep only
   one profile which is capable for compiling/packaging a correct version of
   Sqoop against we can run all of the available test cases.
   - In connection with the goals SQOOP-3052 I do remember that Sowmya and
   Venkat has identified their willingness to do that. If they currently have
   time to do that (after SQOOP-3051 is committed which is about to happen by
   the end of the week) I would be very glad to see that achievement coming by
   their contribution. If by any reason they would not have time for that or
   not interested any more, I'm also willing to find volunteers to do that.
   - Personally I would be very eager to drive through these build related
   changes through the CI system + JIRA + JIRA bots + etc.
   - On the top of these things I'd like to also make one another thing
   happen before the next release, and that would be about to create a proper
   quoting support for MySQL and PostgreSQL as well. If I'm not mistaken the
   Oracle quoting was originated from Sowmya, so it would be also a good
   candidate for her to design+create a generalized and centralized quoting
   mechanism for Sqoop, and have 2-3 different strategies/db (e.g.
   Oracle/MySQL/PostgreSQL, etc.). I would be happy to provide my help (by
   creating the related JIRA tasks, and sharing my insights, and coaching) to
   her, or to anyone willing to do this feature if Sowmya wouldn't have time
   right now to do that.
   - If we're aiming for 1.4.7 I think this scope should be enough.
   - If we're aiming for 1.5. I think we should also include the
   elimination of the com.cloudera classes+packages.
   - I would be also volunteering to drive+own the release and the release
   process.

What do you think about this plan? Could you please share your thoughts?

Many thanks in advance,
Attila

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message