<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Hi,<div><br></div><div><div>First, please note that while COMAD has some promise as a technology for supporting stopping and resuming of workflows, capabilities for doing this with COMAD currently do not exist. It is a (proposed) research area where COMAD is concerned. In short, COMAD does not solve this problem at this time. If you still want to try out COMAD please read on and feel free to contact me for additional information.</div><div><br><div></div></div><div>You can find the COMAD actors and support classes a couple different ways depending on how you are working with Kepler. </div><div><br></div><div>If you are developing with Kepler you can find the source files for the COMAD-related capabilities in Kepler under src/org/nddp. This is where you will find them if you have, say, checked out the Kepler 1.0 release of Kepler from the subversion repository. There are sample workflows and data sets under workflows/nddp.</div><div><br></div><div>A newer version of this code is stored in the part of the subversion repository set aside for Kepler extensions: <a href="https://code.kepler-project.org/code/kepler/modules/">https://code.kepler-project.org/code/kepler/modules/. </a>If you want to build this code the best way to do this for now is to follow the instructions in this <a href="http://pantara.genomecenter.ucdavis.edu:8000/display/Kepler/Quick+Mini-Tutorial+on+the+New+Build+System">tutorial</a> on using a new build system for developing Kepler extensions. The code in the extension area will supplant the code in the main source area for Kepler as soon as we have a way of easily packaging and sharing extensions to Kepler, at which point the procedure will be to download and install Kepler, then download and install the comad extension.</div><div><br></div><div>However, if you just want to try out some sample Kepler workflows based on COMAD, the easiest thing to do is to download and run the <a href="http://daks.ucdavis.edu/kepler-ppod/">Kepler/ppod preview release</a> for OS X. Again, I'll be happy to ask questions about the COMAD capabilities demonstrated in this "preview release". </div><div><br></div><div>You can also read about the pPOD preview release and COMAD in the <a href="https://dev.kepler-project.org/users/whats-new/kepler-newsletter-september-2008">latest Kepler newsletter</a>.</div><div><br></div><div>Cheers,</div><div><br></div><div>Tim</div><div><br></div><div>On Sep 10, 2008, at 8:45 AM, Josep Maria Campanera Alsina wrote:</div><div><div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi all,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I'm back again, I'm extremely interested also in these extension</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">actors that are able to manage the execution of a Kepler workflow.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">But where can I find the COMAD and "Smart rerun" actors? I haven't</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">been able to allocate them in the Kepler repository.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">All the best,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Josep Maria,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Date: Tue, 26 Aug 2008 05:59:27 -0700</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">From: "Bertram Ludaescher" <<a href="mailto:ludaesch@ucdavis.edu">ludaesch@ucdavis.edu</a>></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Subject: Re: [kepler-users] Stop and resume execution...</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">To: "Quentin BEY" <<a href="mailto:quentin.bey@onera.fr">quentin.bey@onera.fr</a>></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Cc: <a href="mailto:kepler-users@ecoinformatics.org">kepler-users@ecoinformatics.org</a></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Message-ID:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span class="Apple-converted-space"> </span><<a href="mailto:657a810a0808260559x305d349ahc64dcdf60e95fd88@mail.gmail.com">657a810a0808260559x305d349ahc64dcdf60e95fd88@mail.gmail.com</a>></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Content-Type: text/plain; charset="iso-8859-1"</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi Quentin:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Interesting question! There are several answers to this.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">First, "knowing which actor was executing" is generally not enough to resume</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">execution:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Consider a workflow executing with a PN (process network) director. Then all</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">actors execute as independent processes (Java threads really), so all are</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">executing simultaneously.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">In contrast, a (sub-)workflow executing under SDF or DDF will be executed</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">within a single thread, so at most one actor is executing at a given time in</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">such a workflow.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">(SDF creates a schedule "statically", i.e., prior to workflow execution,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">while DDF figures out which actors are ready to fire at runtime, then</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">selects one and repeats)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">But what you really need is to maintain the "workflow state" (or some part</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">of it) persistently, so that you can resume a stopped or failed workflow.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">One general way to do this is checkpointing, i.e., writing relevant</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">information out to disk at certain times. While checkpointing can be very</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">costly in general applications, in scientific workflows it can often be</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">easier to do so, since usually components are loosely coupled, all</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">information flow is visible via the channels (unless you do some</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">side-effects outside the model), and actors are often (but not always)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">stateless.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I'm aware of several extensions that allow one to resume Kepler workflows (I</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">think Ptolemy might have further ways):</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-- One system has been called "smart rerun" (e.g. Ilkay Altintas or Dan</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Crawl can point you to it) and allows you to rerun a workflow with modified</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">inputs and/or parameter settings, avoiding to re-execute parts that are</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">"unchanged". I don't recall whether it handles only successful workflow runs</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">(and optimizes their re-execution under change) or also partial (aborted)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">runs.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-- Norbert Podhorszki has developed workflows where actors themselves write</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">out to disk some small information (in his case: remote commands that</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">successfully terminated) which is used upon re-running the workflow to only</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">execute the commands not yet successfully completed previously.<span class="Apple-converted-space"> </span>Call this</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">the "custom checkpointing" solution (instead of a general system extension,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">individual actors or workflows decide what to checkpoint; more work, but it</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">can be more efficient to know what is needed to rerun).</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-- One new director and workflow programming model called COMAD makes</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">visible most if not all of the execution state visible "on the wire" by</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">streaming nested data collections between actors. Like in other approaches,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">the information on the wire can be written to disk and the workflow resumed</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">based on this info.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">All these approaches are based on record information during runtime on disk</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">(sometimes called 'provenance information'), which is then used when</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">resuming the workflow.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">The above options are not the only ones (e.g. Ptolemy probably has</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">additional ways to restart a failed model). Which variant to choose (or</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">which new variant to develop) may depend on, among other things:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-- the size of data flowing through channels (or the availability of</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">persistent ids to large chunks of data)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-- whether actors are stateful or stateless</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-- the director(s) programming/execution model being used</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Bertram</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On Tue, Aug 26, 2008 at 2:22 AM, Quentin BEY <<a href="mailto:quentin.bey@onera.fr">quentin.bey@onera.fr</a>> wrote:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div> <blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi all,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Once again I need help about Kepler's possibilities.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I wonder if we can stop a workflow, quit Kepler, then reopen Kepler and</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">resume the workflow. For instance, a workflow which take long time to</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">execute stops because the computer shutdown (for whatever reason we</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">ignore), if we know which actor was executing is there a simple way to</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">resume execution from this actor?</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks in advance,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Quentin BEY -ONERA- France</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div></blockquote></blockquote></blockquote></div></div></div></body></html>