[kepler-users] Help understanding Kepler paradigm for batch processing images

[Jeremy Douglass]

Bertram, thanks for the suggestion, and for your paper on COMAD. Yes,
"SDC" was my typo of SDF. I've since tried changing the director to
DDF and PN for all my tests in the situation above, although that had
no effect on the type mismatch problem -- Directory Listing produces
an array, and I can't get that into a form consumable by an operation
such as Image Converter.

I read the paper on COMAD with interest. COMAD isn't part of 1.0 right
now, but I look forward to a version that includes it.

-- Jeremy


On Sat, Jul 11, 2009 at 1:09 PM, Bertram Ludaescher<ludaesch at ucdavis.edu> wrote:
>
>
> On Sat, Jul 11, 2009 at 12:51 PM, Jeremy Douglass <jeremydouglass at gmail.com>
> wrote:
>>
>> I'm a new Kepler and struggling to apply a single actor (e.g. Image
>> Converter) to a batch of objects (e.g. a directory of images, as
>> represented by the array generated by Directory Listing).
>>
>> I've read the documentation and searched the forums, and tried
>> modifying examples, but I'm not understanding something paradigmatic,
>> and would appreciate a tip: how do you pipe an array of files through
>> single-file actors?
>>
>> EXAMPLE:
>>
>> In Kepler 1.0.0, 03-ImageDisplay.xml uses an SDC director, and specifies:
>>   Constant Image Filename --> Image Converter --> ImageJ
>
> I think you mean SDF here..
>
>>
>> In an attempt to converter a *directory* of images, I tried adding a
>> Directory Listing actor:
>>   Constant Dirname --> Directory Listing --> Image Converter --> ImageJ
>>
>> ...however this generates an error, as Image Converter will only act
>> on a string, not an array. So I tried
>>   Constant Dirname --> Directory Listing --> Array to Sequence -->
>> Image Converter --> ImageJ
>>
>> ...which almost works if I hand-specified the number of Array elements
>> in the directory ahead of time), but still only processed the first
>> file listed in the string (Picture 1.png) -- I'm assuming it just
>> throws the rest of the string away.
>
>
> SDF has the restriction that the director needs to know a-priori (ie, before
> running the wf) how many tokens an actor consumes and produces on each port
> for each firing. For cases where this is known, SDF is usually a good
> choice.
> However, when processing arrays / lists of varying lengths, you usually
> don't have this information.
> Try DDF instead (or PN if you want actors to execute as independent
> threads).
>>
>> I'm not understanding something about how one processes a list in a
>> Kepler workflow -- I've looked through Array Operation and Iterative
>> Operation actors, and none of them seem to be analogous to an
>> imperative programming loop. Am I using the wrong kind of director?
>> Something else?
>
> See above: DDF might solve your problem.
> There is also a different way to work with nested data collections (of which
> lists are a simple example): using the COMAD director. I'll let the COMAD
> experts and developes say more about how to use this feature (might not be
> part of Kepler 1.0, but in the current developers' version; a new COMAD
> version is also under active development)
>>
>>
>> Thanks for your help.
>
> Also, since you mentioned the paradigma / how-to deal with collections, you
> might want to check out this overview paper:
>
> Scientific workflow design for mere mortals. Timothy McPhillips and Shawn
> Bowers and Daniel Zinn and Bertram Ludaescher (2008) Future Generation
> Computer Systems. Volume 25. Pages: 541-551. (Download)
>
> from the Kepler pubs page:
> https://kepler-project.org/publications?tags=keplerworkflow
>
> Bertram
>
>>
>> BACKGROUND: My larger goal is to adapt a frame based image processing
>> workflow into Kepler (it is currently a combination of python,
>> javascript, and bash scripts). To illustrate, one part of the workflow
>> takes a heterogeneous collection of images (all file types and sizes)
>> and copies them to a new directory as a collection of standardized
>> collection jpegs. Another part of the workflow dumps a movie to a
>> frame directory of jpegs, then runs multiple image measurement and
>> image processing operations (e.g. ImageJ, MATLAB) on each frame and
>> saves the results to a text file.
>> _______________________________________________
>> Kepler-users mailing list
>> Kepler-users at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>
>