[kepler-users] Greetings/newbie questions about Kepler
Matt Jones
jones at nceas.ucsb.edu
Mon Oct 19 20:23:03 PDT 2009
Edward and Christopher covered most of the differences between Kepler and
Ptolemy, especially with respect to the 1.0 release. The current
development version of Kepler that is leading to our 2.0 release includes a
number of additional major differences, some of which include:
1) Module-based extension
- a system for extending Kepler through a well-defined module API that
allows people and projects to extend and package changes to the underlying
system, the GUI, the actors, etc. and the published modules can be loaded
at runtime from within the new (and evolving) Module Manager.
2) A searchable remote repository that can store new actors, workflows,
and their associated metadata in order for people to publish and share
their work
3) A new module for provenance store that allows tracking of the state
of the workflow executions and the data that passes among actors during a
run, and a workflow run manager that uses the provenance store to create a
searchable database of prior executions.
4) A GUI report designer module that allows construction of nicely
formatted reports that organize the outputs and other information about a
workflow run. After execution, these reports can then be viewed in Kepler
and can create PDF versions of the report (which we use for a web-based
reporting system).
5) Extensions to the remote repository and the packaging format (KAR) to
allow workflow execution runs from provenance to be exported into an archive
format, uploaded to the repository, and shared with colleagues.
6) A smenatic tagging module that extends and generalizes the existing
ontology system in Kepler to provide a more intuitive GUI for tagging
actors, workflows, reports, and runs with semantic tags drawn from OWL
ontologies.
Kepler benefits tremendously from the powerful execution engine and
infrastructure that is provided by ptolemy, but over time we've been
improving Kepler by developing features and actors that help streamline the
process of scientific analysis. Hope this helps in your quest for
information. When Kepler 2.0 is updated we'll update the FAQ entry
describing the additional differences between Kepler and Ptolemy.
Matt
On Mon, Oct 19, 2009 at 6:42 AM, Christopher Brooks
<cxh at eecs.berkeley.edu>wrote:
> Hi Tom,
>
> See the Kepler/CORE mission statement at
>
> https://kepler-project.org/users/projects-using-kepler-1/kepler-core-vision-and-mission
>
> For information about the differences between Kepler and Ptolemy,
> http://ptolemy.berkeley.edu/ptolemyII/ptIIfaq.htm#kepler says
> --start--
>
> Ptolemy (http://ptolemy.eecs.berkeley.edu) and The Kepler Project (
> http://www.kepler-project.org/)
> are two separate projects. The Kepler Project FAQ says
>
> What is the difference between Kepler and Ptolemy?
> Roughly speaking, Ptolemy aims at modeling concurrent systems,
> studying system models, various models of computation, etc. as explained
> above.
> In constrast, Kepler aims at execution of scientific workflows (by end
> users
> and/or workflow engineers), inheriting modeling and design capabilities
> including
> the Vergil GUI and workflow scheduling and execution capabilities from
> Ptolemy.
>
> How does Kepler extend Ptolemy?
> Kepler extensions to Ptolemy include an ever increasing number of
> components
> (called actors in Ptolemy terminology) aimed particularly at scientific
> applications,
> e.g., for remote data and metadata access, data transformations, data
> analysis,
> interfacing with legacy applications, web service invocation and
> deployment,
> provenance tracking, etc. Target application areas include bioinformatics,
> cheminformatics,
> ecoinformatics, and geoinformatics workflows among others.
>
> Kepler also inherits from Ptolemy the actor-oriented modeling paradigm
> that
> separates workflow components from the overall workflow orchestration
> (via so-called directors), making components more easily reusable. Through
> actor-oriented
> and hierarchical modeling features built into Ptolemy, Kepler scientific
> workflows can
> operate at very different level of granularity, from low-level "plumbing
> workflows"
> that explictely move data around, start and monitor remote jobs, etc. to
> high-level
> "conceptual workflows" that interlink complex, domain specific data
> analysis steps.
>
> Concerning using Ptolemy actors within Kepler, Norbert Podhorszki writes:
>
> If you find a Ptolemy actor useful, just remember its class name
> (e.g. ptolemy.actor.lib.Ramp). In Kepler, "Tools/Instantiate Component"
> menu allows
> you to type in the class name, which will put an actor instance on your
> canvas just as
> dragging one from the actor tree.
>
> The ptolemy jar is stored in the kepler.jar, so you have access to any
> Ptolemy
> actors and directors.
>
> If you want to use a director not in Kepler tree, you have to use the
> "Tools/Instantiate Attribute" menu. I use it to get a DDF director
> frequently
> (class ptolemy.domains.ddf.kernel.DDFDirector).
>
> I suppose, you can create a kar file for any ptolemy actor just as for
> kepler actors, if you want to add some of them into the Kepler actor
> tree.
> --end--
>
> 2. In my experimentation, the system appears stable and reasonably
>>> efficient.
>>>
>> > But my testing has been on the trivial side. Does the system scale
> well to
> > large sophisticated image processing workflows with many levels of
> composite
> > actors and high volume data streams?
>
>>
>> There are applications with tens of thousands of actors.
>> They require machines with quite a bit of memory, however.
>>
>
> Ptolemy has JAI and JMF actors. There are also actors that pass around Java
> Image
> types and standard matrices of doubles or integers. Using the correct
> token
> type is the key to good performance. Better performance can be had by
> avoiding
> lots of model based conversions between data formats and instead sticking
> with one token
> type and writing a few Java actors. Using a performance analysis tool can
> help.
>
> We are working on various performance improvements that would generate Java
> code
> for hierarchical sub-components, but this work is still in progress.
>
>
> 6. I note you have an actor that provides access to MATLAB.
>>>
>> >> Can the same actor be used for Octave? (see
> http://www.gnu.org/software/octave/).
>
>>
>> I don't think there is anything standard about the API to MATLAB, so
>> I doubt it... But I'm sure one could create an actor for Octave.
>>
>
> The Matlab actor will not work with Octave. Last time I tried to use
> Octave
> (probably 10+ years ago), getting Octave to work required Fortran?. If
> there
> is an C API to Octave, then the Matlab actor could be used as an template.
>
> There are several ways to interface to external programs:
> 1) Use an Exec actor to open a pipe to the external program and pass data.
> This has problems in that it might start up the program each time the actor
> fires. Also, passing data can be tricky
> 2) If there is a C interface, then write a custom actor that sends and
> receives
> data
> 3) Micheal Wetter of LBNL has created a system that uses sockets to connect
> to external programs. This requires writing a little C code, but seems to
> be easier than #2 above. However, I'm not sure if the interface in this
> method
> is as rich as #2.
>
> To me, the biggest issues with interfacing to other programs are:
> - Handling different datatypes can be tricky.
> - Handling error conditions is tricky.
>
>
> _Christopher
>
>
> Edward A. Lee wrote:
>
>>
>> In addition to the packaging and actor library differences,
>> Kepler also has significant UI extensions and infrastructure
>> for searching actor libraries...
>>
>> In some cases, there is functionality that overlaps... E.g.,
>> Kepler has an ontology mechanism, but we have built a somewhat
>> different ontology mechanism into Ptolemy II. This is part of
>> a research project on scalable model engineering. Eventually
>> I hope these two mechanisms will merge, but I can't say with
>> confidence that that will ever happen...
>>
>> Anything else I'm missing?
>>
>> Edward
>>
>> Thomas M. Parris wrote:
>>
>>> Edward,
>>>
>>> Thanks for your prompt reply (over the weekend!). I'll take a closer
>>> look
>>> at the FIR actor and the source for the R and MATLAB wrappers. I've also
>>> been focussing on the Kepler documentation instead of the Ptolemy II
>>> doumentation. It looks like I need to switch focus and look more
>>> specifically at Ptolemy II docs. Is it fair to say the major difference
>>> between Kepler and Ptolemy II is packaging and the actor library?
>>>
>>> -- Tom
>>>
>>>
>>> _______________________________________________
>>> Kepler-users mailing list
>>> Kepler-users at kepler-project.org
>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>>
>> _______________________________________________
>> Kepler-users mailing list
>> Kepler-users at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>
>
> --
> Christopher Brooks (cxh at eecs berkeley edu) University of California
> CHESS Executive Director US Mail: 337 Cory Hall
> Programmer/Analyst CHESS/Ptolemy/Trust Berkeley, CA 94720-1774
> ph: 510.643.9841 fax:510.642.2718 (Office: 545Q Cory)
> home: (F-Tu) 707.665.0131 (W-F) 510.655.5480
>
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nceas.ucsb.edu/kepler/pipermail/kepler-users/attachments/20091019/dc89a840/attachment.html>
More information about the Kepler-users
mailing list