[kepler-dev] startup time

[Aaron Schultz]

As an aside and because I have a few minutes to kill at the moment.  I  
have wanted to reimplement the LibraryIndex code (which is vintage 1.0  
code that uses XML for persistence) and replace it with an SQL  
implementation using the Preorder Tree Traversal Technique since I  
believe it would be faster and easier to maintain.  However, the time  
to do this may run into the couple of weeks range. In either case we  
could ship a prebuilt index with the release which would cut down the  
initial start time for users substantially.

-Aaron

On Oct 9, 2009, at 4:37 PM, Aaron Schultz <aschultz at nceas.ucsb.edu>  
wrote:

> The first time the actor library is built takes longer than  
> subsequent calls to getActorLibrary, that is, if the LibraryIndex  
> class is there for speeding up subsequent calls, however it is not  
> being used at the moment and needs some work and more testing before  
> it gets reinstated.  Part of the slow down is rebuilding that index  
> every time instead of building it once and using it to speed up  
> later library builds.
>
> -Aaron
>
> On Oct 9, 2009, at 4:09 PM, Derik Barseghian <barseghian at nceas.ucsb.edu 
> > wrote:
>
>> Inside generateActorLibrary, it looks like buildFolders is the  
>> expensive call, and more precisely, the first call to the recursive  
>> addFolder method, which I presume is the call that's adding all the  
>> actors to the Actors folder. I'm going to stop poking around here  
>> and wait for Aaron, he wrote this and probably knows if it can be  
>> sped up. If it can't, Sean and Aaron maybe you can find a way to  
>> avoid making this expensive call twice.
>>
>> So, back to the original thread, that's just one of the slow-downs.  
>> Next up is to find out why CacheSearch:rebuild takes so long...
>>
>> Derik
>>
>>
>>> That call makes the default ontology where tags go and makes the  
>>> system aware of that ontology. It's pretty necessary. Ideally it  
>>> would not have to trigger a rebuild of the entire actor library,  
>>> though.
>>>
>>> - Sean
>>>
>>> On Oct 9, 2009, at 3:38 PM, Derik Barseghian wrote:
>>>
>>>> I've taken a look to see why generateActorLibrary is called twice  
>>>> when using wrp, but only once in vanilla. When using wrp the  
>>>> second call occurs because tagging's OntologyCatalog  
>>>> createDefaultOntology calls addOntology which calls buildLibrary  
>>>> which calls generateActorLibrary.
>>>>
>>>> I don't know much about the tagging code. Sean, is that call  
>>>> necessary? Maybe it is...probably the more relevant issue is why  
>>>> does generateActorLibrary take so long, and can it be sped up...
>>>>
>>>> Derik
>>>>
>>>>
>>>> On Oct 9, 2009, at 2:40 PM, Aaron Schultz wrote:
>>>>
>>>>> Yeah could definitely use some optimization.  Does it take that  
>>>>> long with the Kepler suite too?  Is it better on subsequent  
>>>>> startups?  I won't be able to look at it until mon or tue.  Have  
>>>>> no idea why the library would build twice in wrp but doesn't do  
>>>>> it in Kepler suite.  Very strange bugs...
>>>>>
>>>>> -Aaron
>>>>>
>>>>> On Oct 9, 2009, at 12:42 PM, Chad Berkley  
>>>>> <berkley at nceas.ucsb.edu> wrote:
>>>>>
>>>>>> I just confirmed Derik's observations.  My output looks like  
>>>>>> this:
>>>>>>
>>>>>> From a fresh wrp checkout:
>>>>>> ..........
>>>>>> [run] INFO  (org.kepler.objectmanager.cache.CacheSearch:rebuild: 
>>>>>> 161) Rebuilding the search index.
>>>>>> ..........
>>>>>>  [run] INFO  
>>>>>> (org.kepler.objectmanager.library.LibraryManager:generateActorLibrary: 
>>>>>> 1087) Generating Actor Library...
>>>>>> ..........
>>>>>>  [run] INFO  
>>>>>> (org.kepler.objectmanager.library.LibraryManager:generateActorLibrary: 
>>>>>> 1087) Generating Actor Library...
>>>>>> ..........
>>>>>> BUILD SUCCESSFUL
>>>>>> Total time: 3 minutes 49 seconds
>>>>>> berkley at Slickrock build-area$
>>>>>>
>>>>>> 2nd wrp run:
>>>>>> BUILD SUCCESSFUL
>>>>>> Total time: 1 minute 4 seconds
>>>>>> berkley at Slickrock build-area$
>>>>>> * The build still hangs in the same places, but for not as long.
>>>>>>
>>>>>> kepler run:
>>>>>> BUILD SUCCESSFUL
>>>>>> Total time: 28 seconds
>>>>>> berkley at Slickrock build-area$
>>>>>> * I don't see the same log output as above
>>>>>>
>>>>>>
>>>>>> The generateActorLibrary call seems to happen twice.  Each one  
>>>>>> takes about 15-30 seconds to complete.  The CacheSearch:rebuild  
>>>>>> call takes about 30 seconds.
>>>>>>
>>>>>> Aaron, are these the things you are still optimizing?
>>>>>>
>>>>>> chad
>>>>>>
>>>>>>
>>>>>> Derik Barseghian wrote:
>>>>>>> Hey,
>>>>>>> Starting up after a clean-all with the wrp suite seems to be  
>>>>>>> taking much longer now. It gets stuck at "CacheSearch:rebuild: 
>>>>>>> 161) Rebuilding the search index" and then  
>>>>>>> "LibraryManager:generateActorLibrary:1087) Generating Actor  
>>>>>>> Library". It's taking me about 4m 25s to launch.
>>>>>>> I just launched with the kepler suite after a clean-all and it  
>>>>>>> took 2m 21s. Anyone know why the difference?
>>>>>>> (Also, maybe unrelated, but sometimes I'm getting errors when  
>>>>>>> trying to drag out an actor out from the tree, e.g. String  
>>>>>>> Constant...now sure when this is and is not happening...)
>>>>>>> Derik
>>>>>>> _______________________________________________
>>>>>>> Kepler-dev mailing list
>>>>>>> Kepler-dev at kepler-project.org
>>>>>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>>>>> _______________________________________________
>>>>>> Kepler-dev mailing list
>>>>>> Kepler-dev at kepler-project.org
>>>>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>>>
>>>> _______________________________________________
>>>> Kepler-dev mailing list
>>>> Kepler-dev at kepler-project.org
>>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>>
>>
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev