[kepler-dev] Memory Useage

[Bertram Ludaescher]

I agree with Kevin's use of a database. Such a "remote index" is
probably the only way once we have a remote Kepler repository with
(tens of) thousands of actors (loadable on demand of course). 
Think e.g. one actor per R function (per Rpackage!), one per
"Kepler-registered" web service operation etc.

After "quick-search" actors which are not already loaded (given the
above scenario, this will typically be 99% of the actors..) might show
up visually differently from those which are loaded. Once the user
drags and drops the desired actor(s) on the canvas, they are retrieved
from the remote repository and "plugged in".

Nothing for release 1.0 but something that I think can't be avoided as
we scale to thousands of components. 

BTW, is there currently some info out there on what it takes to
1) "register" an actor (with the Kepler repository), and then 
2) "plug-in" a registered actor into a running instance?

Bertram



>>> On Wed, 21 Dec 2005 15:58:35 -0600
>>> Kevin Ruland <kruland at ku.edu> wrote: 
KR> 
KR> What I mean about the database is we can store just enough information 
KR> in there to support the library views and searching.  I don't exactly 
KR> what information would be required but something like:  Name, 
KR> categorization, little icon assignment, lsid, more data?.  I was 
KR> thinking something even lighter weight than the ActorMetadata proxy 
KR> object you mention but in any case it would certainly be smaller than 
KR> the current ActorMetadata.
KR> 
KR> Right now I don't have hard numbers on the total memory requirements for 
KR> each entry in ActorMetadata.  I'll see if I can't convince the standard 
KR> java profiling tools to hand this over to me.
KR> 
KR> Kevin
KR> 
KR> Shawn Bowers wrote:
KR> 
>> I'm not sure I'm really following what Kevin has said below in terms of
>> the database, etc.
>> 
>> But, I just wanted to make a note on keeping actor objects loaded in
>> memory:  First, is there any idea of how large these objects are?
>> Second, to support search, we'd either need to (1) load "proxy" objects
>> for each actor (e.g., the ActorMetadata objects would be a good
>> candidate), or (2) implement some type of mechanism that only loaded
>> (either proxy or the actual actor)  objects that are at any one time shown
>> in the actor library. I'm thinking approach (1) wouldn't be that much
>> savings. Approach (2) would be more scalable but require a lot of work
>> because we'd have to completely rewrite the actor library stuff (and may
>> result in slowdown for navigating and otherwise interacting with the actor
>> library).
>> 
>> However, I'm in general supportive of going the proxy route regardless of
>> this issue, because it is a more elegant way to deal with actors that
>> can't be loaded for whatever reason at runtime.
>> 
>> 
>> -shawn
>> 
>> 
>> On Wed, 21 Dec 2005, Kevin Ruland wrote:
>> 
>> 
>> 
>>> Dan,
>>> 
>>> That would be a great savings.  It would also potentially speed up
>>> startup in the "populated cache" scenario.  Any changes to that system
>>> has to wait until we are back with a full corpus of developers (in
>>> particular Chad).  My opinion is some judicious use of the internal
>>> database would greatly simplify startup and prevent loading all the
>>> actors into memory simultaneously.
>>> 
>>> BTW - you can try my one line hack to ptolemy MoMLParser to see if it
>>> frees up enough memory to execute those workflows.  Christopher will
>>> probably come up with a much better solution but it appeared to work for me.
>>> 
>>> Kevin
>>> 
>>> Dan Higgins wrote:
>>> 
>>> 
>>> 
>>>> Hi All,
>>>> 
>>>   Kevin and Christopher have had several emails on memoery leaks in
>>>> Kepler. Even when we get the leaks fixed, I am seeing another memory
>>>> issue - memory useage.. I am now getting some java 'out of memeory'
>>>> errors on some workflows on a 512M Windows machine. I think we really
>>>> would like to have Kepler run on a machine with 512MB of RAM, so we
>>>> should do some thinking  about reducing memory use. The most obvious
>>>> solution I can think of is to avoid loading all actor into memory at
>>>> program startup.
>>>> 
>>>> Dan
>>>> 
>>>> 
>>>> 
>>>      
>>>> 
>>> _______________________________________________
>>> Kepler-dev mailing list
>>> Kepler-dev at ecoinformatics.org
>>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>>> 
>>> 
>>> 
KR> 
KR> _______________________________________________
KR> Kepler-dev mailing list
KR> Kepler-dev at ecoinformatics.org
KR> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev