[kepler-users] Help understanding Kepler paradigm for batch processing images

[Bertram Ludaescher]

Hi Jeremy:

Can you send the part oft he workflow that creates the problem.
Seems to be a couple of actors only.
Maybe looking at that will help us, help you.

best

Bertram

PS Re. COMAD release plans, I'm cc-ing Tim who should have more info.

On Mon, Jul 13, 2009 at 9:39 AM, Jeremy Douglass
<jeremydouglass at gmail.com>wrote:

> Bertram, thanks for the suggestion, and for your paper on COMAD. Yes,
> "SDC" was my typo of SDF. I've since tried changing the director to
> DDF and PN for all my tests in the situation above, although that had
> no effect on the type mismatch problem -- Directory Listing produces
> an array, and I can't get that into a form consumable by an operation
> such as Image Converter.
>
> I read the paper on COMAD with interest. COMAD isn't part of 1.0 right
> now, but I look forward to a version that includes it.
>
> -- Jeremy
>
>
> On Sat, Jul 11, 2009 at 1:09 PM, Bertram Ludaescher<ludaesch at ucdavis.edu>
> wrote:
> >
> >
> > On Sat, Jul 11, 2009 at 12:51 PM, Jeremy Douglass <
> jeremydouglass at gmail.com>
> > wrote:
> >>
> >> I'm a new Kepler and struggling to apply a single actor (e.g. Image
> >> Converter) to a batch of objects (e.g. a directory of images, as
> >> represented by the array generated by Directory Listing).
> >>
> >> I've read the documentation and searched the forums, and tried
> >> modifying examples, but I'm not understanding something paradigmatic,
> >> and would appreciate a tip: how do you pipe an array of files through
> >> single-file actors?
> >>
> >> EXAMPLE:
> >>
> >> In Kepler 1.0.0, 03-ImageDisplay.xml uses an SDC director, and
> specifies:
> >>   Constant Image Filename --> Image Converter --> ImageJ
> >
> > I think you mean SDF here..
> >
> >>
> >> In an attempt to converter a *directory* of images, I tried adding a
> >> Directory Listing actor:
> >>   Constant Dirname --> Directory Listing --> Image Converter --> ImageJ
> >>
> >> ...however this generates an error, as Image Converter will only act
> >> on a string, not an array. So I tried
> >>   Constant Dirname --> Directory Listing --> Array to Sequence -->
> >> Image Converter --> ImageJ
> >>
> >> ...which almost works if I hand-specified the number of Array elements
> >> in the directory ahead of time), but still only processed the first
> >> file listed in the string (Picture 1.png) -- I'm assuming it just
> >> throws the rest of the string away.
> >
> >
> > SDF has the restriction that the director needs to know a-priori (ie,
> before
> > running the wf) how many tokens an actor consumes and produces on each
> port
> > for each firing. For cases where this is known, SDF is usually a good
> > choice.
> > However, when processing arrays / lists of varying lengths, you usually
> > don't have this information.
> > Try DDF instead (or PN if you want actors to execute as independent
> > threads).
> >>
> >> I'm not understanding something about how one processes a list in a
> >> Kepler workflow -- I've looked through Array Operation and Iterative
> >> Operation actors, and none of them seem to be analogous to an
> >> imperative programming loop. Am I using the wrong kind of director?
> >> Something else?
> >
> > See above: DDF might solve your problem.
> > There is also a different way to work with nested data collections (of
> which
> > lists are a simple example): using the COMAD director. I'll let the COMAD
> > experts and developes say more about how to use this feature (might not
> be
> > part of Kepler 1.0, but in the current developers' version; a new COMAD
> > version is also under active development)
> >>
> >>
> >> Thanks for your help.
> >
> > Also, since you mentioned the paradigma / how-to deal with collections,
> you
> > might want to check out this overview paper:
> >
> > Scientific workflow design for mere mortals. Timothy McPhillips and Shawn
> > Bowers and Daniel Zinn and Bertram Ludaescher (2008) Future Generation
> > Computer Systems. Volume 25. Pages: 541-551. (Download)
> >
> > from the Kepler pubs page:
> > https://kepler-project.org/publications?tags=keplerworkflow
> >
> > Bertram
> >
> >>
> >> BACKGROUND: My larger goal is to adapt a frame based image processing
> >> workflow into Kepler (it is currently a combination of python,
> >> javascript, and bash scripts). To illustrate, one part of the workflow
> >> takes a heterogeneous collection of images (all file types and sizes)
> >> and copies them to a new directory as a collection of standardized
> >> collection jpegs. Another part of the workflow dumps a movie to a
> >> frame directory of jpegs, then runs multiple image measurement and
> >> image processing operations (e.g. ImageJ, MATLAB) on each frame and
> >> saves the results to a text file.
> >> _______________________________________________
> >> Kepler-users mailing list
> >> Kepler-users at kepler-project.org
> >> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
> >
> >
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nceas.ucsb.edu/kepler/pipermail/kepler-users/attachments/20090713/b4a57b04/attachment.html>