[kepler-dev] object manager status

[Chad Berkley]

Hi,

I've finally got the object manager (OM) working and importing ksw files 
into the system.  Note that if you want to try this functionality out, 
you just have to change the basicKeplerLibrary.xml file (in 
configs/ptolemy/configs/kepler/) so that the <property 
name="_alternateLibraryBuilder"...> lines are uncommented and the 
<group>....</group> lines are commented out.  There are instructions 
within that file on what to do.

What happens when you enable the _alternativeLibraryBuilder property is 
that it tells the system to use the specified class to build the actor 
library instead of the static moml files (and momlparser) that reside 
within the configuration.  The class that I currently have setup, 
org.kepler.moml.KSWLibraryBuilder, takes another property, _libraryDir, 
which tells it what directory to look in for ksw files.  You can specify 
multiple directories to search by adding another property called 
"_libraryDirX" where 'X' can be anything.  the name of the property just 
has to start with "_libraryDir."

Once the KSWLibraryBuilder locates all of the ksw files in the given 
directories, it unpacks them and looks at the manifest for the lsid of 
each object in the ksw file.  It then tries to register the object with 
the given lsid into the lsid sub-system.  If there is a conflict, the 
ksw file is skipped and an error message is printed.  the OM supports 
normal LSIDs as well as 'localhost' LSIDs.  the localhost lsids are not 
resolved by the normal LSID resolver, instead they are handled by the OM 
and resolved to a local file.  The class KeplerLSID handles this 
transition.

On startup, a checksum of each ksw file is generated and checked against 
the cached version.  This should force the cached ksw to be updated if 
the ksw file is updated.

Within a KSW file, there are 7 different types of objects.  Each object 
is described by two attributes in the manifest: a type and an lsid.  the 
type must be one of the following:
* data - some sort of data object like an excel file or a text table
* actorMetadata - a moml file describing one actor
* xmlMetadata - xml formatted metadata (other than MoML) like EML or 
Darwin Core
* jar - a jar file containing java executables
* class - a java class file
* nativeLibrary - a binary precompiled native library (like a dll)
* workflow - a MoML workflow

The manifest gives the type and lsid of each file in the ksw, but the 
actor metadata shows how all the objects are interconnected.  Here's an 
example of 'actorMetadata' for the Constant actor (with C style comments 
inserted):
<?xml version="1.0"?>
<entity name="Constant">
//the entityId field gives the lsid for this actorMetadata file
<property name="entityId"  value="urn:lsid:localhost:actor:101:1" 
class="org.kepler.moml.NamedObjId"/>
//the documentation property is reserved for human and/or
//machine readable documentation about this actor
<property name="documentation" 
class="org.kepler.moml.DocumentationAttribute">
A constant actor
</property>
//the class property tells what class implements this actor
<property name="class" value="ptolemy.actor.lib.Const" 
class="ptolemy.kernel.util.StringAttribute">
// the id property within class tells the id of the class
// that implements the actor.  note this can be a java class
// or it can be a MoML class
   <property name="id" value="urn:lsid:localhost:class:101:1" 
class="ptolemy.kernel.util.StringAttribute"/>
</property>
//all ports are explicitly declared
<port name="output">
   <property name="direction" value="output"/>
   <property name="dataType" value="unknown"/>
   <property name="isMultiport" value="false"/>
</port>
<port name="trigger">
   <property name="direction" value="input"/>
   <property name="dataType" value="unknown"/>
   <property name="isMultiport" value="true"/>
</port>
//the semantic type is the link into one or more ontologies for
//subsequent classification within the system
<property name="semanticType0" 
value="urn:lsid:lsid.ecoinformatics.org:onto:1:1#ConstantActor" 
class="org.kepler.sms.SemanticType"/>
//there is another property called 'dependency' which would allows
//you to build a dependency tree for the actor.  objects like
//native libraries can be assigned to the actor here.
//the syntax would look like:
//<property name="dependency0"
//value="urn:lsid:lsid.ecoinformatics.org:dep:101:1"
//class="org.kepler.moml.DependencyAttribute"/>
</entity>

There can be one or more actor metadata files within a ksw file.  Each 
one represents one 'actor' within the system.  Note that two actors 
might be implemented by the same class.

Once the ksw file is read in by the KSWLibraryBuilder and all of the 
lsids are checked and verified for uniqueness, each object in the ksw is 
cached.  The ObjectManager (which implements the actual cache) allows 
easy access to each of the objects via the object's lsid.  You can get 
the object back from the cache as a file, stream or reader.  The OM 
handles getting and caching any remote objects (via the LSIDResolver) 
and you can 'put' local files with localhost lsids with the putLocalFile 
method.

The OM also allows you to get the file handle of a temp file for use in 
processing.  The calling agent never needs to know where or how the file 
is stored (though it can find out), it just calls getTempFile() and is 
returned a file handle.  The temp file also has an lsid and is cached 
for later retrieval.  Temp files are deleted on kepler startup so they 
do not persist through kepler sessions.

You can also iterate through the entire contents of the cache. 
ObjectManager.getCacheItemIterator() returns an Enumeration of all 
objects in the cache.

The LSIDGenerator utility class can be used to create new, unique 
(local) lsids.  It will give you the next object number (for a 
namespace) or the next revision (for an object).  Newly created lsids 
are registered so they won't be used again.  Caution should be used with 
this because the lsids generated are unique locally, but may not be 
remotely.  Additional checks are needed to verify global uniqeness.

My next step is to convert the system to using the OM full time instead 
of having to uncomment the property from the config file.  I need to get 
all of the current actors into ksw form before we can do that.  I'm 
going to work on automating that process next week.  In the meantime, if 
you want to try this with your own ksw file, you can use the 
org.kepler.ksw.KSWCreator command line app to create one.  You can 
either give it all the info it needs on the command line, or run it with 
one command line argument (the working dir) and it'll ask you for the 
info interactively.  Once it creates the ksw file for you, you can just 
stick it into the kepler/ksw directory and have it load.  This should 
work with MoML or Java classes.  The file kepler/ksw/Sinewave.ksw is an 
example of a ksw file that uses a MoML class instead of a Java class.

I'm also working on making this all thread safe.  I'm in the process of 
synchronizing the essential methods (and code blocks) but I've only just 
started.  This is tricky, so when i think i'm done, i'll probably ask 
one or more of you to take a look at it to make sure i haven't missed 
anything.

Just wanted to let everyone know what the status of the OM is.  Let me 
know if you have any comments or suggestions.

chad