[kepler-dev] recommendations for automated actor tests

Sun Mar 12 11:26:54 PST 2006

Hi Tim,
You suggestions look very comprehensive.

A note about terminology:

The term "Actor tests" implies unit tests (actors being an atomic unit
of execution), which test workflows are not.  A test system that
includes the code under test usually called a system test or
integration test.  System tests will catch different bugs than unit
tests.  Personally, I prefer to emphasize unit tests over system tests
because it is easier to comprehensively test error conditions and
boundary conditions.  Usually one writes unit tests using something
like JUnit.  I do think that both unit tests and system tests are
necessary, I just focus on unit tests over system tests. I think
Edward prefers test workflows over unit tests.  I'm fine with "actor
tests" though.

One aspect that is missing is that I don't see any code coverage in
the plan.  Locally we use an obsolete code coverage product from Sun,
I'm sure there are other products out there.  Code coverage will
tell us which actors have no tests and which methods and blocks are
untested.  100% code coverage does not mean that a program is tested,
but without some form of code coverage it is difficult to have any
idea of how well tested a piece of code is.  Perhaps the Kepler
nightly build has code coverage in it already?

Another aspect that needs to be addressed is handling graphical tests
and actors.  Currently, the Ptolemy II tests are non-graphical.
There are two reasons: 
 1) We want to be able to ship non-graphical systems.
 2) Graphical testing that includes regressive tests is difficult.
    It is easy to smoke test a system that pops up a window, it is
    hard to say that the window that was popped up is correct.

How do you intend to test graphical systems?  Are tests allowed to
pop up windows?  

_Christopher 

--------

    All,

    I volunteered to oversee some aspects of actor testing in preparation  
    for the 1.0 release.  In particular, I will be inspecting the actor  
    tests listed in lib/test-workflows.lst to verify that they test the  
    actors we include in the release.  I indicate my recommendations for  
    writing actor tests below.

    Note that while we have sometimes used the term "test workflows," we  
    are actually trying to test particular actors here, not workflows.   
    (We are using workflows to test actors.)  For this reason, I refer to  
    workflows meant to test actors as "actor tests".

    Please let me know if you have any comments, corrections, or  
    alternative suggestions.  I will post an updated version of the  
    material below to the Kepler wiki once everyone has had a chance to  
    comment.

    Thanks!

    Tim

    *** Motivation ***

    Why include automated tests for Kepler actors?  Two reasons have been  
    suggested:

    1.  Existence of a working test for an actor will be the criteria for  
    including an actor in the 1.0 release.
    2.  We want tests for all actors included in the 1.0 release.

    Comments:  The first motivation implies that it is practical to write  
    automated tests for all actors in 1.0, while the second motivation  
    might imply that we will deliver well-tested actors exclusively.   
    While neither of these implications seem entirely probable, we'll do  
    the best we can.  Note also that there are additional good reasons to  
    have actor tests, but we will not focus on these here.

    *** Assumptions ***

    I have made several assumptions about our actor tests and testing  
    framework.  They are the basis for the recommendations that follow.

    1.  We want fully automated tests only.   In particular, the nightly  
    build must run each actor test, and the nightly build report must  
    accurately report test success or failure on an actor-by-actor basis.
    2.  We assume that all actors provided with Ptolemy II are well- 
    tested.  Consequently, we do not require automated tests for Ptolemy  
    II actors in Kepler. (Contributing tests for these actors is  
    encouraged, however!)
    3.  Actor tests should be runnable not just on the nightly build  
    system, but also on developers' systems.
    4.  Collectively, the actor tests should exercise the key function(s)  
    provided by each actor.
    5.  It should be clear what actor(s) each automated test is intended  
    to test.
    6.  It should be relatively straightforward to determine from a test  
    failure which actor failed.
    7.  Actor tests are not expected to test Kepler code outside of actors.
    8.  Successfully testing an actor with a particular director does not  
    imply that the actor works with any other director.
    9.  We want to compose, document, and maintain actor tests using the  
    Kepler GUI.

    Comments:  The first assumption excludes the possibility of fully  
    testing actors that generate graphical output or depend on user  
    interaction.  We will need a different criterion for including such  
    actors in Kepler 1.0.  The second assumption also requires a distinct  
    mechanism for deciding what actors to include in 1.0.

    *** Recommendations ***

    1.  Focus each actor test on testing a single actor.  There may be  
    more than one test for a particular actor.  The name of the test  
    should indicate which actor is being tested as well as the specific  
    nature of the test.  I suggest a name such as "ArrayLength_SDF.moml"  
    to imply that only one test under SDF is required, and a name like  
    "ArrayAppend_SDF_two_channels.moml" to indicate one of several  
    possible tests for an actor.

    2. Actor tests may be included anywhere in the kepler repository.   
    The relative path to each actor test (e.g., workflows/test/test- 
    ecogrid-eml-gce-data.xml) should be included in the file lib/test- 
    workflows.lst.  This is the file used by the nightly build script to  
    determine what actor tests to run.   Maintain an alphabetical order  
    of the test (i.e., actor) names (not the relative paths).

    3.  Before including a test in the nightly build, test it locally  
    using the ptexecute ant target in build.xml, i.e., by issuing a  
    command such as:

               ant ptexecute -Dworkflow=workflows/eco/Elk_Wolf.xml

    4.  Any local data files required for the test should be included in  
    the kepler repository.  Within the actor test, specify the path to  
    any such files relative to the kepler project directory.

    5.  Where possible, use the Test, NonStrictTest, and TypeTest actors  
    to validate actor output (see documentation for these actors).

    6.  Use the minimum number of actors and connections required to test  
    a particular function of an actor.

    7.  Document each actor test internally (e.g., using a TextAttribute  
    a.k.a. "Annotation").  Indicate what function of the actor is being  
    tested, how its inputs are prepared, how its outputs are validated,  
    etc.  Strive to make it easy to determine what has gone wrong when a  
    test fails.

    8.  Document what external applications are required to successfully  
    run the test, and how to acquire installers for these applications.   
    Make sure that required applications are installed on the nightly  
    build system before committing the test to the repository.

    9.  Include your name and e-mail address in the documentation for  
    each test.  I will be examining the actor tests periodically and  
    contacting authors of tests that I cannot understand.

    *** Additional open questions  to consider ***

    1.  How many Kepler actors can be adequately tested using this approach?
    2.  What additional actors or tools might make it easier to test  
    Kepler actors?

    _______________________________________________
    Kepler-dev mailing list
    Kepler-dev at ecoinformatics.org
    http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
--------