[kepler-dev] Documentation of demo workflows for Kepler Alpha 9 release

[Christopher Brooks]

Hi Zhijie,
The Ptolemy II testing framework is documented in 
$PTII/doc/coding/testing.htm.
There are two components, unit tests written in Tcl and model tests that run 
models
in the test/auto directories.  Both the unit and model tests are written in 
a way that
does not require a graphical display head.  For the model tests, we use a 
MoML
filter that removes graphical components - actually, it substitutes 
non-graphical
components for graphical components.  The model tests don't have plotters or
user input, instead they use Test or NonStrictTest actors as output.  User 
input
is not something we are simulating at this point.  To test demos, I usually 
create
a model test that has Test actors.  The problem with this is that the actual 
model
and the demo are now two different files and thus instantly out of date.  It 
would
be nice if we could have a PlotTest actor that could be a plotter when run 
in
a regular way by the user and a Test actor when run as a test.  I've looked 
in to
various GUI test schemes but I've not looked that recently.  The problems 
I've seen
with the GUI test schemes are that they are either very expensive to 
purchase
or else expensive to maintain, where trivial changes result in lots of 
retraining of
the test.  Another aspect of GUI tests is that a smoke test - does the test 
run
without crashing - is not worth much because it does not test the 
correctness
of the run.  In addition, tests need to have code coverage - without code 
coverage,
we have no idea as to how complete our testing is.  Unit tests result in 
much
better line by line coverage than model (aka system) tests.

I'd be up for finding a GUI test scheme that meets our needs.  The Tcl tests 
are
something that I use, but JUnit has much greater market share and is 
probably
preferred - this even though Tcl tests are interpreted and loosely typed, 
which is
much more convenient that JUnit's compiled and strongly typed testing 
framework.

_Christopher
----- Original Message ----- 
From: "Zhijie Guan" <guan at sdsc.edu>
To: "Christopher Brooks" <cxh at eecs.berkeley.edu>
Cc: <Kepler-dev at ecoinformatics.org>
Sent: Saturday, February 18, 2006 6:51 PM
Subject: Re: [kepler-dev] Documentation of demo workflows for Kepler Alpha 9 
release


> Hi, Christopher,
>
> Thanks for sharing the Ptolemy experience with us. It is really important 
> to have a systematic way for workflow documentations. The way described in 
> my email is only a temporary proposal to record related info about 
> workflows. I think we still need more discussion on the approach for 
> Kepler workflow documentations in the near future. I'd like to use the 
> proposed way to collect information for other developers (like myself) to 
> test existing workflows, and for the end users to understand how to run 
> workflows.
>
> BTW, would you please give me some idea about automatically testing models 
> in Ptolemy? I'd like to integrate the workflow testing in Kepler nightly 
> build. But I don't know how to simulate the user input in the automatic 
> testing.
>
> Enjoy your vacation!
>
> Zhijie
>
>
> On Sat, 18 Feb 2006, Christopher Brooks wrote:
>
>> Hi Zhijie,
>> .doc is probably not the right extension since under Windows .doc means
>> Microsoft Word.
>> Perhaps it would be better to put this information in a .htm file
>> with <h2>...</h2> sections for each of the questions?
>> That way, anyone can easily edit the file by double clicking on it.
>> Perhaps the file should be called ModelNameDoc.htm?
>>
>> In Ptolemy II, we found it very helpful to follow naming conventions
>> for files.  So, a demo called "Spectrum" is a .xml file called
>> demo/Spectrum/Spectrum.xml.  For applets, the applet would
>> be called demo/Spectrum/Spectrum.htm.  Perhaps the
>> additional documentation would be in a file called
>> demo/Spectrum/SpectrumDoc.htm?
>>
>> We required demo names to follow the Java class naming
>> conventions (alphanumeric only) so that we can easily generate
>> code for demos.
>>
>> The advantage of a strict naming convention is that it makes it
>> easy to refer to other demos without struggling with the contents
>> of each directory.
>>
>> _Christopher
>>
>> ----- Original Message ----- From: "Zhijie Guan" <guan at sdsc.edu>
>> To: <Kepler-dev at ecoinformatics.org>
>> Sent: Friday, February 17, 2006 5:39 PM
>> Subject: [kepler-dev] Documentation of demo workflows for Kepler Alpha 9 
>> release
>>
>>
>>> Dear workflow authors,
>>>
>>> To make sure all the workflows included in Kepler Alpha 9 release can be
>>> run successfully, Ilkay and I need your help to prepare a documentation
>>> for each workflow on how to run them.
>>>
>>> The documentations should be checked in into the CVS repository along 
>>> with
>>> the workflows. We hope each workflow could have its own doc just like 
>>> each
>>> actor has its own doc. Since currently we do not have common-agreed
>>> framework about how to document workflows, I suggest we use plain text
>>> files to record workflow related info. The document should be stored in
>>> the same directory with the workflow, have the same name with the
>>> workflow, but with a file extension ".doc". We can easily transform the
>>> plain text file to the doc format if we have a doc framework for 
>>> workflows
>>> in the future.
>>>
>>> The doc for a workflow should answer the following questions:
>>>
>>> 1. What's the goal/purpose of this workflow?
>>>
>>> 2. Who are the authors?
>>>
>>> 3. How to run this workflow?
>>> 3.1 What OS can this workflow be run on? (Windows/Linux/Unix/Mac)
>>> 3.2 What additional software are needed?
>>> 3.3 Does the user need to do any settings? (like the path of the 
>>> temporary
>>> file, the number of iteration times, etc)
>>> 3.4 Does the user need any account/password?
>>> 3.5 What database/web services/web-based applications are used in the
>>> workflow? Are they available usually?
>>> 3.6 Is there any other things that the user need to do to run this
>>> workflow?
>>>
>>> 4. What does the user expect to see as the result of the workflow?
>>>
>>>
>>> So here I suggest we write the doc according to the following format
>>> (shown as a doc for workflow "RecIDCM3.xml").
>>>
>>>
>>> RecIDCM3.doc
>>> ================================================
>>> Goal: Simulate RecIDCM3 algorithm with the CORBA services provided by 
>>> the
>>> CIPRES SDK.
>>>
>>> Author(s): Zhijie Guan, Terri Liebowitz, Alex Borchers (contact:
>>> guan at sdsc.edu)
>>>
>>> Compatible OS:
>>> Mac (Yes. tested)
>>> Windows (No. CIPRES SDK is not compatible with windows)
>>> Linux/Unix (No. CIPRES SDK is not compatible with Linux/Unix)
>>>
>>> Additional Software: CIPRES SDK (which is not included in Kepler) should
>>> be installed in $KEPLER/lib/jar/cipres
>>>
>>> Setting(s): The user needs to choose the input NEXUS file from the
>>> FileChooser dialog
>>>
>>> Account/Password: N.A.
>>>
>>> Database: N.A.
>>>
>>> Web Services: N.A.
>>>
>>> Web-based Applications: N.A.
>>>
>>> Comments: N.A.
>>>
>>> Results: The user should see two phylogenetic trees shown in two ATV
>>> windows. One tree comes from the input NEXUS file. The other is the
>>> optimal tree improved by RecIDCM3 algorithm.
>>> ===================================================
>>>
>>> Any suggestions/comments/opinions are highly appreciated!
>>>
>>>
>>>
>>> Regards,
>>>
>>> Zhijie Guan
>>> _______________________________________________
>>> Kepler-dev mailing list
>>> Kepler-dev at ecoinformatics.org
>>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>>>
>>
>