[kepler-dev] recommendations for automated actor tests

[Timothy McPhillips]

All,

I volunteered to oversee some aspects of actor testing in preparation  
for the 1.0 release.  In particular, I will be inspecting the actor  
tests listed in lib/test-workflows.lst to verify that they test the  
actors we include in the release.  I indicate my recommendations for  
writing actor tests below.

Note that while we have sometimes used the term "test workflows," we  
are actually trying to test particular actors here, not workflows.   
(We are using workflows to test actors.)  For this reason, I refer to  
workflows meant to test actors as "actor tests".

Please let me know if you have any comments, corrections, or  
alternative suggestions.  I will post an updated version of the  
material below to the Kepler wiki once everyone has had a chance to  
comment.

Thanks!

Tim



*** Motivation ***

Why include automated tests for Kepler actors?  Two reasons have been  
suggested:

1.  Existence of a working test for an actor will be the criteria for  
including an actor in the 1.0 release.
2.  We want tests for all actors included in the 1.0 release.

Comments:  The first motivation implies that it is practical to write  
automated tests for all actors in 1.0, while the second motivation  
might imply that we will deliver well-tested actors exclusively.   
While neither of these implications seem entirely probable, we'll do  
the best we can.  Note also that there are additional good reasons to  
have actor tests, but we will not focus on these here.



*** Assumptions ***

I have made several assumptions about our actor tests and testing  
framework.  They are the basis for the recommendations that follow.

1.  We want fully automated tests only.   In particular, the nightly  
build must run each actor test, and the nightly build report must  
accurately report test success or failure on an actor-by-actor basis.
2.  We assume that all actors provided with Ptolemy II are well- 
tested.  Consequently, we do not require automated tests for Ptolemy  
II actors in Kepler. (Contributing tests for these actors is  
encouraged, however!)
3.  Actor tests should be runnable not just on the nightly build  
system, but also on developers' systems.
4.  Collectively, the actor tests should exercise the key function(s)  
provided by each actor.
5.  It should be clear what actor(s) each automated test is intended  
to test.
6.  It should be relatively straightforward to determine from a test  
failure which actor failed.
7.  Actor tests are not expected to test Kepler code outside of actors.
8.  Successfully testing an actor with a particular director does not  
imply that the actor works with any other director.
9.  We want to compose, document, and maintain actor tests using the  
Kepler GUI.

Comments:  The first assumption excludes the possibility of fully  
testing actors that generate graphical output or depend on user  
interaction.  We will need a different criterion for including such  
actors in Kepler 1.0.  The second assumption also requires a distinct  
mechanism for deciding what actors to include in 1.0.



*** Recommendations ***

1.  Focus each actor test on testing a single actor.  There may be  
more than one test for a particular actor.  The name of the test  
should indicate which actor is being tested as well as the specific  
nature of the test.  I suggest a name such as "ArrayLength_SDF.moml"  
to imply that only one test under SDF is required, and a name like  
"ArrayAppend_SDF_two_channels.moml" to indicate one of several  
possible tests for an actor.

2. Actor tests may be included anywhere in the kepler repository.   
The relative path to each actor test (e.g., workflows/test/test- 
ecogrid-eml-gce-data.xml) should be included in the file lib/test- 
workflows.lst.  This is the file used by the nightly build script to  
determine what actor tests to run.   Maintain an alphabetical order  
of the test (i.e., actor) names (not the relative paths).

3.  Before including a test in the nightly build, test it locally  
using the ptexecute ant target in build.xml, i.e., by issuing a  
command such as:

           ant ptexecute -Dworkflow=workflows/eco/Elk_Wolf.xml

4.  Any local data files required for the test should be included in  
the kepler repository.  Within the actor test, specify the path to  
any such files relative to the kepler project directory.

5.  Where possible, use the Test, NonStrictTest, and TypeTest actors  
to validate actor output (see documentation for these actors).

6.  Use the minimum number of actors and connections required to test  
a particular function of an actor.

7.  Document each actor test internally (e.g., using a TextAttribute  
a.k.a. "Annotation").  Indicate what function of the actor is being  
tested, how its inputs are prepared, how its outputs are validated,  
etc.  Strive to make it easy to determine what has gone wrong when a  
test fails.

8.  Document what external applications are required to successfully  
run the test, and how to acquire installers for these applications.   
Make sure that required applications are installed on the nightly  
build system before committing the test to the repository.

9.  Include your name and e-mail address in the documentation for  
each test.  I will be examining the actor tests periodically and  
contacting authors of tests that I cannot understand.



*** Additional open questions  to consider ***

1.  How many Kepler actors can be adequately tested using this approach?
2.  What additional actors or tools might make it easier to test  
Kepler actors?