[kepler-users] Help understanding Kepler paradigm for batch processing images

[Bertram Ludaescher]

On Sat, Jul 11, 2009 at 12:51 PM, Jeremy Douglass
<jeremydouglass at gmail.com>wrote:

> I'm a new Kepler and struggling to apply a single actor (e.g. Image
> Converter) to a batch of objects (e.g. a directory of images, as
> represented by the array generated by Directory Listing).
>
> I've read the documentation and searched the forums, and tried
> modifying examples, but I'm not understanding something paradigmatic,
> and would appreciate a tip: how do you pipe an array of files through
> single-file actors?
>
> EXAMPLE:
>
> In Kepler 1.0.0, 03-ImageDisplay.xml uses an SDC director, and specifies:
>   Constant Image Filename --> Image Converter --> ImageJ
>

I think you mean SDF here..


>
> In an attempt to converter a *directory* of images, I tried adding a
> Directory Listing actor:
>   Constant Dirname --> Directory Listing --> Image Converter --> ImageJ
>
> ...however this generates an error, as Image Converter will only act
> on a string, not an array. So I tried
>   Constant Dirname --> Directory Listing --> Array to Sequence -->
> Image Converter --> ImageJ
>
> ...which almost works if I hand-specified the number of Array elements
> in the directory ahead of time), but still only processed the first
> file listed in the string (Picture 1.png) -- I'm assuming it just
> throws the rest of the string away.
>


SDF has the restriction that the director needs to know a-priori (ie, before
running the wf) how many tokens an actor consumes and produces on each port
for each firing. For cases where this is known, SDF is usually a good
choice.
However, when processing arrays / lists of varying lengths, you usually
don't have this information.
Try DDF instead (or PN if you want actors to execute as independent
threads).

>
> I'm not understanding something about how one processes a list in a
> Kepler workflow -- I've looked through Array Operation and Iterative
> Operation actors, and none of them seem to be analogous to an
> imperative programming loop. Am I using the wrong kind of director?
> Something else?


See above: DDF might solve your problem.
There is also a different way to work with nested data collections (of which
lists are a simple example): using the COMAD director. I'll let the COMAD
experts and developes say more about how to use this feature (might not be
part of Kepler 1.0, but in the current developers' version; a new COMAD
version is also under active development)

>
>
> Thanks for your help.
>

Also, since you mentioned the paradigma / how-to deal with collections, you
might want to check out this overview paper:


   - *Scientific workflow design for mere mortals.* Timothy McPhillips and
   Shawn Bowers and Daniel Zinn and Bertram Ludaescher (2008) Future
   Generation Computer Systems. Volume 25. Pages: 541-551.
(Download<http://dx.doi.org/10.1016/j.future.2008.06.013>)



from the Kepler pubs page:
https://kepler-project.org/publications?tags=keplerworkflow

Bertram


>
> BACKGROUND: My larger goal is to adapt a frame based image processing
> workflow into Kepler (it is currently a combination of python,
> javascript, and bash scripts). To illustrate, one part of the workflow
> takes a heterogeneous collection of images (all file types and sizes)
> and copies them to a new directory as a collection of standardized
> collection jpegs. Another part of the workflow dumps a movie to a
> frame directory of jpegs, then runs multiple image measurement and
> image processing operations (e.g. ImageJ, MATLAB) on each frame and
> saves the results to a text file.
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nceas.ucsb.edu/kepler/pipermail/kepler-users/attachments/20090711/f71929ac/attachment.html>