[kepler-dev] Writing Files inside workflows

[Dan Higgins]

Hi All,
    There are currently several actors in Kepler for writing text files: 
e.g. 'org.geon.FileWrite' and 'org.resurgence.actor.TextFileWriter'.  
These actors apparently do the same thing (write a string to a file) but 
the details of how they do it can apparently make a big difference! Let 
me explain.

    The GARP workflow "garpModel_ASC_withData.xml" is an example of 
getting species locations from Digir and then running GARP. The species 
locations (longitude, latitude pairs) are concatenated into a string and 
then written to a file that is an input to the GarpPresampleLayers 
actor. Originally, the workflow used a FileWriter actor to write the 
input long,lat file. This seemed to work fine. Recently, however, I 
tested this workflow again and had all sorts of problems (Kepler 
crashes, strange error messages, etc.) In trying to figure out the 
problem, I discovered that the Digir sources have apparently be updated 
and are now returning more locations than before. The location file is 
thus longer (and takes more time to write). To summarize a whole bunch 
of debugging, the problem was apparently the FileWriter! FileWriter 
writes the file in the postfire method and then doesn't close the file 
until the wrapup method (in its parent LineWriter class). But I was 
using it in the middle of a workflow; I really want it to completely 
write and close the file before the next actor starts reading the file. 
It thus looks like there are some threading/timing issues with using 
FileWriter.

    Well, it turns out that TextFileWriter does all of its work writing 
a file INSIDE the 'fire' method. If I replace the FileWriter with the 
resurgence TextFileWriter, all the problems I was having disappear!

    So the lesson appears to be that internal details can be very important!

Dan

-- 
*******************************************************************
Dan Higgins                                  higgins at nceas.ucsb.edu
http://www.nceas.ucsb.edu/    Ph: 805-893-5127
National Center for Ecological Analysis and Synthesis (NCEAS) 
Marine Science Building - Room 3405
Santa Barbara, CA 93195
*******************************************************************