[kepler-dev] object manager status

[Bing Zhu]

I am not aware of your new design. The code I put was based on
our design. I was also "mad" and struggling with DataCacheManager
which is so hard to clean up and to accommodate our design of cache
items. That's why I sent several mails regarding this file.

I am glad that you have found new way to handle with ksw files in cache.

I have only one concern, which is actually I tried to find a nice solution
for,
is that the cache items used in current 'DataCacheManager.java' are used
in other places (previous code to use this Data Cache) which might consider
those cached objects (classes, libs, etc) from ksw is cached query data from
Ecogrid.

Bing






-----Original Message-----
From: Chad Berkley [mailto:berkley at nceas.ucsb.edu]
Sent: Tuesday, August 23, 2005 9:36 AM
To: bzhu at sdsc.edu
Cc: Matt Jones; Reagan Moore; Arcot Rajasekar; Kepler-Dev
Subject: Re: [kepler-dev] object manager status


Hi Bing,

Yeah, I removed those files because they were no longer needed with the
redesign that I did.  I'm sorry that I haven't gotten around to changing
the wiki to look like the new design, but I've been coding this
furiously to get it done by the time i leave for vacation on saturday.
Basically, the way it now works is that the ObjectManager wraps the
existing DataCacheManager to do caching.  It uses the existing
DataCacheObject class to hold cache items.  The way I am storing the
type information is within an attribute in the DataCacheObject.  The
DataCacheObject already has the functionality to add arbitrary
attributes onto any cache item, thus i'm storing the lsid and the type
within the object (the lsid is actually the name of the object so all
objects are unique).  This does away with the need to have a seperate
class for each type of object in the cache.

You can see how the different object types get assigned to the cache
objects in the method ObjectManager.handleKSWItem().  It basically reads
the type from inside the ksw file, the caches each item based on that
type (using the static ints that are defined within ObjectManager).

This is a very simple system that makes use of a lot of pre-existing
code.  I've cleaned up the DataCacheManager code considerably, though it
still needs some work.  I'm hesitant to do more to it as the
ObjectManager is working well and I haven't broken anything else in
kepler and I don't really want to.

If you want to see more on how the ObjectManager works, check out the
ObjectManagerTest in tests/test/org/kepler/objectmanager/.

chad

Bing Zhu wrote:
> Hi Chad,
>
> Just had a fresh checkout of kepler code. And found even those cache item
> classes
> had been removed. Just wonder how the cache item system works without
having
> these classes
> as you mentioned in your mail regarding 7 types of objects in cache place.
> Note that
> the current DataCacheManager.getCacheItem() or
> DataCacheManager.getCacheItems()
> returns only DataCacheObject or DataCacheObjects which has nothing to do
> with our
> designed cache items as I pointed out in my mails before. Let me know.
>
> Bing
>
>
>
>
>
> -----Original Message-----
> From: kepler-dev-bounces at ecoinformatics.org
> [mailto:kepler-dev-bounces at ecoinformatics.org]On Behalf Of Chad Berkley
> Sent: Friday, August 19, 2005 2:34 PM
> To: Kepler-Dev
> Subject: [kepler-dev] object manager status
>
>
> Hi,
>
> I've finally got the object manager (OM) working and importing ksw files
> into the system.  Note that if you want to try this functionality out,
> you just have to change the basicKeplerLibrary.xml file (in
> configs/ptolemy/configs/kepler/) so that the <property
> name="_alternateLibraryBuilder"...> lines are uncommented and the
> <group>....</group> lines are commented out.  There are instructions
> within that file on what to do.
>
> What happens when you enable the _alternativeLibraryBuilder property is
> that it tells the system to use the specified class to build the actor
> library instead of the static moml files (and momlparser) that reside
> within the configuration.  The class that I currently have setup,
> org.kepler.moml.KSWLibraryBuilder, takes another property, _libraryDir,
> which tells it what directory to look in for ksw files.  You can specify
> multiple directories to search by adding another property called
> "_libraryDirX" where 'X' can be anything.  the name of the property just
> has to start with "_libraryDir."
>
> Once the KSWLibraryBuilder locates all of the ksw files in the given
> directories, it unpacks them and looks at the manifest for the lsid of
> each object in the ksw file.  It then tries to register the object with
> the given lsid into the lsid sub-system.  If there is a conflict, the
> ksw file is skipped and an error message is printed.  the OM supports
> normal LSIDs as well as 'localhost' LSIDs.  the localhost lsids are not
> resolved by the normal LSID resolver, instead they are handled by the OM
> and resolved to a local file.  The class KeplerLSID handles this
> transition.
>
> On startup, a checksum of each ksw file is generated and checked against
> the cached version.  This should force the cached ksw to be updated if
> the ksw file is updated.
>
> Within a KSW file, there are 7 different types of objects.  Each object
> is described by two attributes in the manifest: a type and an lsid.  the
> type must be one of the following:
> * data - some sort of data object like an excel file or a text table
> * actorMetadata - a moml file describing one actor
> * xmlMetadata - xml formatted metadata (other than MoML) like EML or
> Darwin Core
> * jar - a jar file containing java executables
> * class - a java class file
> * nativeLibrary - a binary precompiled native library (like a dll)
> * workflow - a MoML workflow
>
> The manifest gives the type and lsid of each file in the ksw, but the
> actor metadata shows how all the objects are interconnected.  Here's an
> example of 'actorMetadata' for the Constant actor (with C style comments
> inserted):
> <?xml version="1.0"?>
> <entity name="Constant">
> //the entityId field gives the lsid for this actorMetadata file
> <property name="entityId"  value="urn:lsid:localhost:actor:101:1"
> class="org.kepler.moml.NamedObjId"/>
> //the documentation property is reserved for human and/or
> //machine readable documentation about this actor
> <property name="documentation"
> class="org.kepler.moml.DocumentationAttribute">
> A constant actor
> </property>
> //the class property tells what class implements this actor
> <property name="class" value="ptolemy.actor.lib.Const"
> class="ptolemy.kernel.util.StringAttribute">
> // the id property within class tells the id of the class
> // that implements the actor.  note this can be a java class
> // or it can be a MoML class
>    <property name="id" value="urn:lsid:localhost:class:101:1"
> class="ptolemy.kernel.util.StringAttribute"/>
> </property>
> //all ports are explicitly declared
> <port name="output">
>    <property name="direction" value="output"/>
>    <property name="dataType" value="unknown"/>
>    <property name="isMultiport" value="false"/>
> </port>
> <port name="trigger">
>    <property name="direction" value="input"/>
>    <property name="dataType" value="unknown"/>
>    <property name="isMultiport" value="true"/>
> </port>
> //the semantic type is the link into one or more ontologies for
> //subsequent classification within the system
> <property name="semanticType0"
> value="urn:lsid:lsid.ecoinformatics.org:onto:1:1#ConstantActor"
> class="org.kepler.sms.SemanticType"/>
> //there is another property called 'dependency' which would allows
> //you to build a dependency tree for the actor.  objects like
> //native libraries can be assigned to the actor here.
> //the syntax would look like:
> //<property name="dependency0"
> //value="urn:lsid:lsid.ecoinformatics.org:dep:101:1"
> //class="org.kepler.moml.DependencyAttribute"/>
> </entity>
>
> There can be one or more actor metadata files within a ksw file.  Each
> one represents one 'actor' within the system.  Note that two actors
> might be implemented by the same class.
>
> Once the ksw file is read in by the KSWLibraryBuilder and all of the
> lsids are checked and verified for uniqueness, each object in the ksw is
> cached.  The ObjectManager (which implements the actual cache) allows
> easy access to each of the objects via the object's lsid.  You can get
> the object back from the cache as a file, stream or reader.  The OM
> handles getting and caching any remote objects (via the LSIDResolver)
> and you can 'put' local files with localhost lsids with the putLocalFile
> method.
>
> The OM also allows you to get the file handle of a temp file for use in
> processing.  The calling agent never needs to know where or how the file
> is stored (though it can find out), it just calls getTempFile() and is
> returned a file handle.  The temp file also has an lsid and is cached
> for later retrieval.  Temp files are deleted on kepler startup so they
> do not persist through kepler sessions.
>
> You can also iterate through the entire contents of the cache.
> ObjectManager.getCacheItemIterator() returns an Enumeration of all
> objects in the cache.
>
> The LSIDGenerator utility class can be used to create new, unique
> (local) lsids.  It will give you the next object number (for a
> namespace) or the next revision (for an object).  Newly created lsids
> are registered so they won't be used again.  Caution should be used with
> this because the lsids generated are unique locally, but may not be
> remotely.  Additional checks are needed to verify global uniqeness.
>
> My next step is to convert the system to using the OM full time instead
> of having to uncomment the property from the config file.  I need to get
> all of the current actors into ksw form before we can do that.  I'm
> going to work on automating that process next week.  In the meantime, if
> you want to try this with your own ksw file, you can use the
> org.kepler.ksw.KSWCreator command line app to create one.  You can
> either give it all the info it needs on the command line, or run it with
> one command line argument (the working dir) and it'll ask you for the
> info interactively.  Once it creates the ksw file for you, you can just
> stick it into the kepler/ksw directory and have it load.  This should
> work with MoML or Java classes.  The file kepler/ksw/Sinewave.ksw is an
> example of a ksw file that uses a MoML class instead of a Java class.
>
> I'm also working on making this all thread safe.  I'm in the process of
> synchronizing the essential methods (and code blocks) but I've only just
> started.  This is tricky, so when i think i'm done, i'll probably ask
> one or more of you to take a look at it to make sure i haven't missed
> anything.
>
> Just wanted to let everyone know what the status of the OM is.  Let me
> know if you have any comments or suggestions.
>
> chad
>
>
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at ecoinformatics.org
> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>