[kepler-dev] Re: Nightly Builds
Christopher Hylands Brooks
cxh at eecs.berkeley.edu
Thu Mar 18 11:34:47 PST 2004
Yes, "make fast" runs javac on a per directory basis.
Looks like an interesting paper, we could do a study group!
I almost included the text below in my original email about Nightly
Builds, but it seemed off topic.
The executive summary is that we could probably solve the issues below
and convert Ptolemy II to Ant. This would probably take 2-4 weeks of
my time, which precludes doing it before the release at the end of
April.
I think Ant is a good thing, and if I was starting out from scratch, I
would consider using Ant. We should train people in using the state
of the art tools, and Ant is one of them.
The negative side of converting to Ant is that it would be disruptive,
and that change for the sake of change is not always a good thing.
There are plenty of interesting tasks, is converting to Ant one of
them?
Anyway, below are my half baked thoughts.
....
Ptolemy II uses makefiles to build partly to make it easier to ship
releases.
A while ago I poked around with Ant, and it had two main features:
1) It was platform independent
For us, we use Cygwin under Windows, which solves the platform
independence issue.
With Ant or Cygwin, there is still a bootstrapping issue, where the
user needs to tell the system where to find the Java compiler and
look for optional programs. We use configure to find these things.
When I looked at Ant, it looked difficult to optionally include
directories and configure the location of optional packages. Ant
has likely worked out some of these issues since I looked at it.
I need two steps: a configuration step and a build step
2) It was not make.
The Ant page used to say that Ant was created because make was
confusing. I think this is partly a not-invented-here attitude.
I've seen other people create build systems that replicate the
make functionality and it seems like lots of effort with little
value added. Initially I believe that Ant did not have
dependencies, but now it does(?)
At the time, for me, the kicker was that I did not see converting the
makefiles to Ant as providing much added value.
I think this has changed somewhat with the emergence of Eclipse and
the acceptance of Eclipse within the Ptolemy group.
If I was starting a new project today, I would strongly consider using
Ant instead of make, but I would need to address some of the issues
below.
1) Configuration
We need a way of having the user tell the system where the components
are. Either we use configure or else some other program.
If we use a java program, then we need to get the user to
run the program, which can be tricky (where's java?)
configure is not exactly easy to use either
2) Excluding directories
I'm somewhat more familiar with Eclipse than Ant, but I think
a similar issue comes up in Ant. Eclipse, like Ant, does blind
javac *.java compiles.
In Eclipse to exclude a directory, we need to add it to the .classpath
file. We might want to exclude a directory because the developer
considers it not ready for use, or because configure could not find a
necessary component.
The problem here is that there is one Eclipse .classpath file for the
project, and we need to adjust it for each directory we want to
exclude.
Ptolemy II makefile have that level of control in the individual
makefile. If I want to exclude a directory, in the parent directory,
I just make sure that in the makefile the DIRS variable does not
include the directory I want to exclude.
3) Managing the content of a directory
Another area is that I want to have fairly tight control over what
files are shipped. The makefiles should have each file listed in
JSRCS or someother variable. Running "make checkjunk" will list
files and directories that for some reason are not listed in the
makefile. I run "make checkjunk" and either remove the files listed,
add them to the makefile or add them to the list of files that
are excluded from the release.
It is amazing the amount of flotsam and jetsam that end up being added
accidentally and never being removed.
4) Creating jar files
Also, each makefile usually can create a jar file. If there are
subdirectories, then the parent jar file may contain jar files from
the subdirectory.
In otherwords if I have
foo
foo.jar
bar
bar.jar
biz
biz.jar
bof
bof.jar
Then foo.jar could contain the files in bar.jar and bof.jar, but
not biz.jar because biz is not shipping or something.
This is a little confusing, but the high level summary is that
I need to be able to build jar files that contain some or all of the
files in any jar files in children directories.
Basically, I need a manifest in each directory that contains what
subdirectories are to be compiled in, what subdirectories have jar
files of interest and what files are to be shipped.
Probably Ant can do this for me, but when I looked at it, it was
tricky.
5) Dependency issues
I think Ant used to not do a good job with dependencies.
There are times that one wants to avoid doing something (say running
javadoc) unless the derived files have changed.
-Christopher
--------
An interesting epistle on the downsides of recursive make:
http://aegis.sourceforge.net/auug97.pdf
This paper claims that you can get the same behavior as recursive make
using includes and local build rules.
I think the javac thing is still a problem though, because you can overrun
the command line doing javac *.java
Steve
At 11:11 AM 3/18/2004 -0800, Stephen Andrew Neuendorffer wrote:
>make fast basically runs one javac for each directory... But you still
>have a javac process and a recursive make process for every directory.
>Given a large number of directories, that's still alot of processes...
>
>At 10:06 AM 3/18/2004 -0900, Matt Jones wrote:
>>Christopher,
>>
>>So, even using "make fast" the build is far slower using "make" than
>>"ant". On my P4 3.06Ghz PII with 1GB ram, these are the stats I get:
>>
>>[jones at snow ptII3.0.2]$ time make clean fast
>>real 6m48.208s
>>user 2m0.610s
>>sys 0m8.580s
>>
>>[jones at snow kepler]$ time ant -f build-ptolemy.xml clean install
>>real 0m23.757s
>>user 0m16.060s
>>sys 0m1.160s
>>
>>Granted, this isn't a totally fair comparison, because the make does more
>>stuff than the ant build. But the ant build does everything *needed* in
>>a normal edit/compile/debug cycle, and in this context the difference
>>between 23 seconds and over six minutes is huge. My hunch is that make
>>is lauching javac separately for every java file, which incurs huge
>>per-file overhead, while ant does its compilation of all java files
>>within a single javac invocation. But that's just a hunch on my part as
>>I haven;t looked into it in detail. And its not just your makefile -- my
>>experience has been that compiling java files using make is slow under
>>all projects where I've seen it used.
>>
>>Thanks for your insights into why this might be,
>>Matt
>>
>>Christopher Hylands Brooks wrote:
>>>[I changed the Subject line]
>>>Matt wrote:
>>>
>>>>[I (Christopher) wrote: ]
>>>>
>>>>>* How are nightly builds and tests handled right now? Forgive my
>>>>>ignorance on this, it might be apparent from the code, I just have
>>>>>not looked.
>>>>
>>>>We don't have a nightly build. We should get one :) I know your
>>>>existing makefile permits this. Somtime I would like to discuss the
>>>>Makefile with you and try to figure out why it builds so much more
>>>>slowly than the ant build that I wrote. Then it would make more sense
>>>>to use it directly for the nightly build.
>>>
>>>BTW - "cd $PTII; make fast" should be much faster than "cd $PTII; make".
>>>The $PTII/adm/gen-3.1/makefile is what I use to run the nightly build.
>>>It is a very gross file, I don't recommend looking at it.
>>>The reason it is gross is that it works under Cygwin and Solaris, and
>>>it does multiple builds on remote machines.
>>>However, what it does is conceptually simple:
>>>1. In a directory that has a preexisting Ptolemy II CVS tree, do
>>> cd $PTII
>>> cvs update -Pd
>>> make clean fast
>>> and run the tests.
>>> In the nightly build, the tests are listed in a strange order to
>>> ferret out package dependencies, but in principle
>>> (cd $PTII; make tests) should work.
>>>2. Build a release by excluding things we don't ship, make a few
>>> modifications to configurations (exclude references to things
>>> that do not ship)
>>>3. Untar the release elsewhere, build (with a different java and make)
>>> and run tests.
>>>4. Run the very long codegen tests and send email about it to only a
>>>few developers.
>>>
>>>So, no matter how you build, you can still do step 1 above.
>>>BTW - If you want to quickly build Ptolemy as part of that, you can
>>>do 'make clean fast'
>>>The other steps are also not particularly make-centric.
>>>I strongly encourage you to set up a nightly build, it is a big win.
>>>Our nightly build is available at
>>>http://chess.eecs.berkeley.edu/ptexternal/nightly/index.htm
>>>http://chess.eecs.berkeley.edu/ptexternal/nightly/coverage.html
>>>is a large table that shows the code coverage for the Java classes.
>>>
>>>BTW - With regard to JUnit vs. Jacl, JUnit is much more of an industry
>>>wide standard, so going with it is appropriate.
>>>The one advantage of using Jacl, a 100% Java implementation of a
>>>subset of Tcl) is that Jacl is interpreted, which makes writing tests
>>>a little easier. I find it is easier to create small test fragments
>>>by typing at a command line prompt and then piece together a larger
>>>test file.
>>>There are many more cool JUnit tools out there than there are Jacl
>>>tools.
>>>-Christopher
>>
>>--
>>-------------------------------------------------------------------
>>Matt Jones jones at nceas.ucsb.edu
>>http://www.nceas.ucsb.edu/ Fax: 425-920-2439 Ph: 907-789-0496
>>National Center for Ecological Analysis and Synthesis (NCEAS)
>>University of California Santa Barbara
>>Interested in ecological informatics? http://www.ecoinformatics.org
>>-------------------------------------------------------------------
>>_______________________________________________
>>kepler-dev mailing list
>>kepler-dev at ecoinformatics.org
>>http://www.ecoinformatics.org/mailman/listinfo/kepler-dev
>
>
>_______________________________________________
>kepler-dev mailing list
>kepler-dev at ecoinformatics.org
>http://www.ecoinformatics.org/mailman/listinfo/kepler-dev
--------
More information about the Kepler-dev
mailing list