[kepler-dev] [Bug 4091] - Make it so the build system can use a "blessed" version of Ptolemy.

[bugzilla-daemon at ecoinformatics.org]

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=4091





------- Comment #1 from cxh at eecs.berkeley.edu  2009-05-26 07:40 -------
We should use the term "stable" to refer to revisions of Ptolemy and Kepler
that are likely to build for users.  A true stable version usually requires
developer time to validate the revisions necessary.  If developer time is
available, in the perfect world, there would be an acceptance test such as:
 - Build Kepler and Ptolemy from a clean checkout
 - Start Kepler 
 - Run a particular model
Other things to try:
 - Build a simple model

The minimum set of acceptance tests for a stable version should be clearly
documented so that users of stable versions know what they are getting.

If developer time is not available, then we could try automatically
designating particular versions as stable, but certain types of failures 
will slip through.
There are several different levels of stability of a particular Ptolemy build:
1) Does Ptolemy and Kepler build properly?  A continuous build system
would help detect problems here.  
Ptolemy does not have a continuous build system, Kepler's continuous build
system does not send useful email.
One idea would be to have a mailing list that included people who made
changes within the last N days and mail them the continuous build info.

If the build fails, then obviously something is wrong and the revisions
of both Kepler and Ptolemy are not stable.

2) Tests should be run and the results evaluated.  Unfortunately, the tricky
part is that Kepler will work fine if some tests fail.  Probably the thing
to do is to develop a few simple system tests that must pass if a particular
revision is to be considered stable.  The system tests should be both
non-graphical and graphical.  These tests could invoke models and test
the known good results.  We need better info about running the Kepler tests.

3) UI tests would help indicate the overall stability.  Common problems include
missing actors and problems building under Eclipse.  Testing for these
automatically would be difficult, but are worth a shot.