openoffice-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre Fischer <>
Subject Re: Build breaker and clean build
Date Fri, 12 Apr 2013 07:27:16 GMT
On 11.04.2013 18:43, Andrew Rist wrote:
> On 4/11/2013 12:06 AM, Andre Fischer wrote:
>> On 10.04.2013 18:57, Andrew Rist wrote:
>>> On 4/10/2013 1:33 AM, Herbert Dürr wrote:
>>>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>>>> tonight we had a build breaker in the windows build: a slot id 
>>>>> that is
>>>>> used in SW had been removed in SVX.  The reference in SW had also 
>>>>> been
>>>>> removed, so this change should not be a problem.
>>>>> But the windows build is still not a full build. Therefore the old SW
>>>>> slot header files where used and the build broke.
>>>>> There is an easy fix for situations like this: a clean build.
>>>> Incremental build are known to have problems thats why I suggested 
>>>> [1] to default to a clean build. That didn't receive consensus 
>>>> though and indeed there are good reasons against it:
>>>> The incremental build both tests the dependency system and it 
>>>> reduces the load when building significantly.
>>>> On the already strained buildbot this means a factor of almost five 
>>>> improvement as clean build takes about 4.5h whereas an incremental 
>>>> build takes only 0.5-1.0h.
>>>> Andrew even had to reschedule the snapshot build away from the 
>>>> weekly clean build because the buildbot load is a real problem.
>>>> [1]
>>>> [2]
>>> Just to add here, that there are also issues with a clean build. The 
>>> clean build fails with some frequency on hung jobs and requires 
>>> manual attention.
>> That is one more reason to have more frequent clean builds so that we 
>> can find the cause of these problems.  They are not restricted to the 
>> build bot.   Others are affected, too.  I would have tried to fix 
>> this, but I am not able to reproduce this problem on my local machine.
> Sorry, I think I was unclear there.  Due to the complexity of our 
> build process and the interaction with the buildbot, there is a 
> reasonably high incidence of false positive failures on clean builds.  
> The Windows build ends up with hung processes and throws an 
> exception.  If we were to switch to clean builds we could expect 
> several false positives a week, which would require manual 
> intervention.  We have tests of clean builds daily on the linux boxes, 
> so in terms of coverage of the entire AOO buildbot setup, we have full 
> builds covered.   I see the fact that some of our builds are 
> incremental and some are clean, as a feature, not as a shortcoming.

OK, I understand.  I see two problems with this approach:

1. We will not find  the build problems when we continue to hide them 
with workarounds.  Fixing them would benefit others as well, not just 
the build bots.

2. The task of the build bots is not only to verify that our source code 
is buildable but also to provide download sets for developers and QA.

>>> In reality, breaking changes that require a clean build are pretty 
>>> rare.  For me, the clean build on the weekend and incremental during 
>>> the week seems to be a good compromise.
>> I am not sure about that.   Besides, it is sometimes a bit difficult 
>> to judge 'how incompatible' a change really is. Change a slot 
>> definition in SVX and its use in SW.  Do we need anything more? With 
>> a clean build we are on the safe side.
> It's a trade off.  How many false positives to you get, and how much 
> manual intervention does the system require.   What I'm trying to 
> communicate is that my experience with 'this buildbot setup' suggests 
> that the current approach requires less of my time to keep healthy, 
> and produces less false positives.

Ah, again I understand.  It is a matter of pragmatism.  I can accept 
that.  I and I think all others on this list are thankful for your work 
in this area.  And maybe we can help to improve the current state.  Is 
there any documentation, a Wiki page maybe, to get an overview?

Maybe we can rededicate one of the daily builds of one of the branches 
to debugging our windows build problems?  Add a step to the build setup 
to apply patches that provide us with more debug info?

>> Besides, the clean-build-on-weekend policy would require us to hold 
>> all incompatible changes until Friday, or live with a broken build 
>> during the week.
>> I really thing that we need a better solution.  A switch for marking 
>> a change as incompatible and that would be interpreted by the build 
>> bot would be the absolute minimum.  But even that would call for 
>> trouble.  At Sun we have been there and it did not really work so well.
> This doesn't require waiting until the weekend, it requires a manual 
> clean run which can be kicked off easily.  (I'm happy to show you, or 
> hit up Herbert - he has access, too)

It would be good to have this knowledge written down somewhere for easy 
reference.  Especially if it really remains our only way to get clean 

>   I don't disagree with your general argument, I just see different 
> trade-offs, and I consider this type failure to have a fairly trivial 
> recovery (kicking off a manual clean build).

Again, I don't think that this is a trivial solution.

First I have to detect that my changes are incompatible.  That is 
sometimes easy but sometimes it is not.  If it is not easy then there is 
even the possibility that the incremental build will complete 
successful.  But the result might show some very subtle errors that we 
may or may not find before a release.
I would rather not spend any time on the analysis.

Then we have to coordinate the triggering of incompatible builds. We 
have developers working around the world at every time of the day.  It 
might be better to have an incompatible build to start a a fixed time 
then have developer A trigger a clean build and developer B, who is one 
hour late, has to do that again right after that. This might flood the 
build servers with AOO builds.


> Andrew
>>>>> This may become important in the coming weeks when we have to fix 
>>>>> some
>>>>> bugs in the sidebar (which is about to be merged back into 
>>>>> trunk).  The
>>>>> sidebar is implemented in several modules.  Without a clean windows
>>>>> build we will run into build breakers very regularly.
>>>> It is possible to force a clean build manually.
>>> I'm cleaning it up now and kicking off a build.
>> Thanks for taking care of this one.
>> -Andre
>>> Andrew
>>>> Herbert
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail:
>>>> For additional commands, e-mail:
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message