openoffice-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre Fischer <awf....@gmail.com>
Subject Re: Make x Dmake x Build (Was: Re: [Error] Compile AOO on Debian amd64)
Date Mon, 22 Apr 2013 07:58:27 GMT
On 19.04.2013 15:09, janI wrote:
> On 19 April 2013 14:40, Andre Fischer<awf.aoo@gmail.com>  wrote:
>
>> On 19.04.2013 10:02, janI wrote:
>>
>>> On 19 April 2013 07:56, Jürgen Schmidt<jogischmidt@gmail.com>  wrote:
>>>
>>>   On 4/19/13 12:33 AM, Kay Schenk wrote:
>>>>> On Thu, Apr 18, 2013 at 2:11 PM, janI<jani@apache.org>  wrote:
>>>>>
>>>>>   On 18 April 2013 22:38, Kay Schenk<kay.schenk@gmail.com>  wrote:
>>>>>>   On Thu, Apr 18, 2013 at 5:17 AM, janI<jani@apache.org>  wrote:
>>>>>>>   On 18 April 2013 14:08, Claudio Filho<filhocf@gmail.com>
 wrote:
>>>>>>>>   Hi
>>>>>>>>> 2013/4/18 Oliver-Rainer Wittmann<orwittmann@googlemail.com>:
>>>>>>>>>
>>>>>>>>>> But regarding the removed Slot FN_PROPERTY_WRAP_DLG
perform a clean
>>>>>>>>>>
>>>>>>>>> build of
>>>>>>>>>
>>>>>>>>>> module sw:
>>>>>>>>>> - cd sw
>>>>>>>>>> - make clean
>>>>>>>>>> - build
>>>>>>>>>>
>>>>>>>>> Oliver, sorry by my newbie ask, but... we don't use more
dmake?
>>>>>>>>>
>>>>>>>>> If i understood correctly, "build" is a perl script that
calls all
>>>>>>>>> modules, building in order of dependence, entering in
each one,
>>>>>>>>> calling Dmake to compile and delivering all files where
need.
>>>>>>>>>
>>>>>>>>>   correct.
>>>>>>>>   I saw some "makefile" files and many more "makefile.mk",
where i
>>>>>>>> think
>>>>>>> that one is for Make and other is to Dmake. I see it in wiki
too, for
>>>>>>>>> build parts.
>>>>>>>>>
>>>>>>>>>   again correct.
>>>>>>>> Problem is that some of the modules have been moved away
from dmake
>>>>>>>> to
>>>>>>>> "gbuild", so right now it is a mix (and not a very smart
one).
>>>>>>>>
>>>>>>>>   Jan --
>>>>>>> This last comment "not a very smart one" is interesting. Do you
care
>>>>>>> to
>>>>>>> elaborate?
>>>>>>>
>>>>>>>   I have to watch more carefully what I write, someone is actually
>>>>> reading it
>>>>> :-)
>>>>>>   yes, we do that sometimes! :)
>>>>>   I am deep in the building system at the moment with my l10n work, and
>>>>> what
>>>>> we have now in trunk is approx 2/3 orignal dmake (that btw also seem
to
>>>>>> have at least 2 generations) and 1/3 gbuild, this combination does
a
>>>>>>
>>>>> good
>>>>> job of confusing anyone who tries to understand the system. Just to make
>>>>>> things worse, the gbuild part is split in as many files as possible.
>>>>>>
>>>>>> So I should have written "dont try to understand it, just accept
it",
>>>>>> actually someone else in here said something similar to me a couple
of
>>>>>> month ago.
>>>>>>
>>>>>>   Exactly that is the problem, the work on gbuild is not yet finished
>>>> and
>>>> we have a mix of gnu make and dmake. The gbuild is the outcome of an
>>>> analysis how to improve the build system. I believe it is not bad and
>>>> our friends from LO have more or less finished the work but I also
>>>> believe that it is too complex and can be done much simpler.
>>>>
>>>> It's nearly impossible to debug and not easy to understand. It tried to
>>>> hide the complexity from single makefiles in the modules and introduced
>>>> soe kind of new scripting language based on gnu make. I am not sure if
>>>> that was the best approach.
>>>>
>>>>   As I wrote earlier, the gbuild team have done a fantastic job analyzing
>>> our
>>> system.
>>>
>>> I am sure there is something wrong the approach, when you have a chain of
>>> 5
>>> makefiles (one include the other)...so to me the structure itself is very
>>> complex.
>>>
>> I personally think this improves what little readability gbuild has.  If
>> you know that you want to fix a bug when building a library then you just
>> have to look at LinkTarget.mk (the 'base class') or Library.mk (derived
>> class with code that does not apply to static libraries).  For platform
>> specific things you look at platform/<platform>.mk.  I would even think
>> about splitting up AllLangResTarget.mk into files for SrsPartMergeTarget,
>> SrsPartTarget, SrsTarget, ResTarget, and AllLangResTarget (one 'class' per
>> file, like in Java).
>> I don't know if having all this in a single makefile would improve
>> maintainability of gbuild.
>>
> I am sure a single makefile would not really improve maintainability. But
> not going through 5 files, would speed up build time, so somewhere in
> between.

I think that the time it takes to read the makefiles is negligible. All 
files under solenv/gbuild have ca 7700 lines in total.  The dependency 
files under solver/workdir/Dep/CxxObject have ca 38000 lines.  The rest 
of the office has a lot more.  From experiments in this area I would 
guess that reading the 7700 lines would take around 100ms, hardly 
noticeable.


> Without going to much in detail, I see a big difference in how you specify
> the makefiles and what is actually running. In the project I worked with,
> we had configure, generate makefiles, I think aoo could benefit from that.

I totally agree.   I think that cmake is doing something similar.

>
>
>>   There is a reason both for the scripting language and the complex
>>> structure. Both dmake and gbuild tries to solve the whole world and cook
>>> coffee at the same time.
>>>
>>> What I mean is, there are no separation between a couple of logical steps:
>>> - Clean/create directories and other items needed for the actual make.
>>> This
>>> happens on the go and everytime you build. In my world we should have a
>>> "make prepare" that I call ONCE after svn co, and the real build should
>>> assume all directories are present.
>>>
>> Maybe we have to distinguish between (hm, how do I put this into english
>> words?) information or knowledge on one hand and process or running actual
>> commands on the other hand.  The information in case of "make clean" is
>> that eg compiling a file <name>.cxx produces a file <name>.obj.  The
>> important part is that <name>.obj is an automatically created file that can
>> be re-created at any time.  So, the 'clean' target depends on <name>.obj
>> but not on <name>.cxx.  The action in this example is that all dependencies
>> of the 'clean' target are deleted.
>>
>>
>> The separation between these two exists in gbuild but is not absolute.
>>   Much of the action, the so called commands, are defined in the
>> solenv/platform/<platform>.mk files while the information, mostly the
>> dependencies between files, are defined in solenv/*.mk.  But you are right,
>> that some of the simpler actions, like deleting files for 'make clean' or
>> the creation of directories before eg compiling files into them, is not
>> cleanly separated.
>>
> I just had a rebuild (not even complete build) after updating trunk..mkdir
> was called 431 times (remark I built with a lot of languages) and every
> time the directory existed.

I see your point.  But lets assume that you have just edited one or two 
files and want to build the module to update its libraries, a common 
task.  If the preparation step is separated from the actual build, then 
one of two things can happen:

1. The preparation step checks all directories that are needed for 
building the module, regardless of whether they are affected by the change.

2. You somehow have to transfer dependency information from the building 
step to the preparation step.


Why not instead add dependencies on the directories?  Make should take 
care to evaluate a dependency only once (I hope but I don't know if that 
is true) and thus only try once to create a directory.

>
>> I don't know how a "make prepare" could work without defining dependencies
>> between files (and directories) in two places.  And I really mean that I
>> don't know, not that I think that this is a bad idea.
>>
>>   - Intermodule dependencies. Every module checks at every build if the
>>> modules it depends has been built. In my world, module makefile should
>>> only
>>> do the module, and a makefile in main takes care of the interdependencies.
>>>
>> That would reintroduce (or not fix) several problems:
>>
>> - Inter-module dependencies as we have them today have to be specified
>> manually.  That is error-prone as is demonstrated by some of the build bot
>> problems.  In contrast Inter-file dependencies can be extracted
>> automatically from header files.  I think that every compiler that we use
>> can do that.
>>
> Yes...but you anyhow specify to the linker what you want to link together,
> so information is available.

Right.  And this should be the only source of information to derive 
dependencies from.

>> - When the build system detects that one file in module A has been
>> modified and as consequence module B has to be built because B depends on
>> A, then every file in A and every file in B has to be built.  In reality
>> this is hardly ever necessary.  I don't know if there are files that with
>> inter-file dependencies would cause this worst case.
>>
> No, that was not what I meant....when a file in module A is changed, module
> A is rebuilt (only the changed files), that leads to either a new header
> file or a new library. The central makefile notes that module A is rebuilt,
> and calls the makefiles for all modules depending on A. HOWEVER these
> makefiles will only compile the sources using a changed header file, and
> link libraries/binaries using a changed library. So you see you get the
> best of two worlds, or in other words the central makefile is only
> responsible for invoking the module makefiles.

You are right, this could work.
Over the weekend I looked into our dependency files, that are generated 
automatically by compilers or mkdepend.  Basically all the information 
for your approach are already there.  It should already work today that 
I change an exported header in SVX, then build SD and exactly those 
files files are compiled that depend directly or indirectly on the 
modified SVX header.  I tried this and it works.
Then why do we still need clean builds?  There are a lot of dependencies 
missing, eg those on SDI files (<module>/sdi/*.sdi), those on IDL files 
and so on.  If we added them then the current build system might already 
do what you propose.

But I still think that a file based approach would be better and could 
be a lot faster because it can parallelize the build on a finer granularity.

>
>> - At the moment a developer has to know whether a change he or she makes
>> is incompatible, ie requires files in other modules to be re-built.  This
>> is sometimes hard to find out and, just to be on the safe side, you have to
>> start an incompatible/clean build.  With inter-file dependencies, the build
>> system knows what has to be rebuild and rebuilds only the affected files.
>>
> You have a very important point here !! I would have inter-file
> dependencies in the module makefiles, but assume that the external parts
> have been created/compiled before this makefile is activated.
>
>
>> But of course there are drawbacks.  One is the increased amount of
>> dependencies that have to be created, read with every build, and checked.
>>   This certainly takes more time than to resolve inter-module dependencies.
>>   Especially when you build the whole office, eg after a fresh checkout from
>> SVN, then inter-module dependencies are probably a bit faster.  But if you
>> change one or just a small number of files (a typical use case in the life
>> of an AOO developer) and there are (potential) incompatible changes then I
>> would always prefer inter-file dependencies.
>>
> Actually I prefer a combination, as written above...but I think it will be
> clearer when I get some examples written.
>
>
>> One example from my current work.  I am currently fixing bugs in the
>> sidebar.  Its implementation is located in sfx2, svx, sw, sc, sd with
>> configuration files in officecfg.  If I make an incompatible change in
>> sfx2/source/sidebar then only sidebare related files in svx, sw, sc and sd
>> are affected. Should be done in a couple of minutes.  But with our current
>> inter-module dependencies I have to do a full clean build from sfx2.  That
>> takes about one hour.
>>
> This  is pure hell today, and that is something we must address in a new
> system.
>
> Another item is the number of processes created, today the compiler is
> invoked pr file, that is no good...in windows starting the compiler is
> expensive, so when it is started we should compile all changed (and only
> changed) files in one step.

Good idea.

-Andre

>
>
>>> I have tried the LO gbuild, and it is not particulary fast either.
>>>
>> Interesting.  Did you do a full build or changed only a few files?
>>
> Both, I wanted to see if some if the problems I see have been solved. I
> find they have done quite a good job.
>
> But try one little thing, do a build and the do another build. The second
> build should do nothing (apart from checking makefiles)  that is not the
> case (at least in the version I tested).
>
> rgds
> jan I.
>
>> Regards,
>> Andre
>>
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail:dev-unsubscribe@openoffice.**apache.org<dev-unsubscribe@openoffice.apache.org>
>> For additional commands, e-mail:dev-help@openoffice.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org


Mime
View raw message