flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chesnay Schepler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8819) Rework travis script to use build stages
Date Sat, 03 Mar 2018 21:37:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384864#comment-16384864

Chesnay Schepler commented on FLINK-8819:

A full green build can be found [here|https://travis-ci.org/zentol/flink/builds/348724165].

The total build times hasn't changed (+-5%); all the time we save by avoiding the redundant
compiles is eaten up by the additional overhead of stages, like the downloading/unpacking
of the cache).

The job times have generally improved however, the "misc" profile, which currently starts
scratching the 50m limit, now only takes 27m. This may be enough justification to do the switch,
even though we didn't accomplish the desired goal.

> Rework travis script to use build stages
> ----------------------------------------
>                 Key: FLINK-8819
>                 URL: https://issues.apache.org/jira/browse/FLINK-8819
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Build System, Travis
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>            Priority: Trivial
> This issue is for tracking efforts to rework our Travis scripts to use [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after another. This
implies that we can define dependencies between jobs, in contrast to our existing jobs that
have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple jobs.
> The main benefit here is that we no longer have to compile modules multiple times, which
would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing build-artifacts
from one stage to the next. According to this [issue|https://github.com/travis-ci/beta-features/issues/28]
it is on their to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by either using
the Travis cache or some other external storage. The cache solution would work by setting
up a cached directory (just like the mvn cache) and creating build-scope directories within
containing the artifacts (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
>  We can clean things up if
>  * our script fails
>  * the last stage succeeds.
> We can *not* clean things up if
>  * the build is canceled
>  * travis fails the build due to a timeout or similar
> as apparently there is [no way to run a script at the end of a build|https://github.com/travis-ci/travis-ci/issues/4221].
> Thus we would either have to periodically clear the cache, or encode more information
into the cached files that would allow _other_ builds to clean up stale date. (For example
the build number or date).

This message was sent by Atlassian JIRA

View raw message