[kepler-dev] Kepler can only be invoked once

Dan Higgins higgins at nceas.ucsb.edu
Thu May 17 10:08:25 PDT 2007


Hi Lisong,
    The class 'org.kepler.gui.KeplerInitializer' takes care of 
initialization of the internal database (see 
http://www.kepler-project.org/Wiki.jsp?page=KeplerStartup )
But I suspect that simply removing the cache setup there will cause 
other problems with KeplerApplication because the building of the actor 
list on the left uses the cache system. MoMLSimpleApplication avoids 
this problem because it never displays the main Kepler screen; it just 
executes a workflow.

Dan Higgins
NCEAS

----

Lisong Sun wrote:
> Seems like MoMLSimpleApplication does not setup the cache system hence 
> made it possible to run multiple instances.
> Is there anyway I can make the KeplerApplication class not using the 
> cache as well? We can write a subclass of KeplerApplication and invoke 
> from there if that is what it takes. Wonder which piece of code that 
> is actually responsible for the setup of cache.
>
> Lisong
>
> Dan Higgins wrote:
>>
>> Hi Lisong,
>>     One way of getting around the data cache problem is to use
>> 'momlexecute' to run kepler workflows. I have put together a new page on
>> the Kepler wiki describing different ways of launching Kepler from the
>> command line - see
>>
>> http://www.kepler-project.org/Wiki.jsp?page=ExecutingKepler
>>
>> Dan Higgins - NCEAS
>>
>> Lisong Sun wrote:
>> > Thank your all for the helpful info. About the multiple running 
>> problem,
>> > we probably won't care much about data caching. So what does it 
>> take in
>> > order to make it working. For example, if the data cache directory is
>> > controlled by an environment variable, we can generate unique temp dir
>> > for each running instance and pass in the path for kepler to use. Or
>> > simply turn off the data caching totally if that is possible.
>> >
>> > Lisong
>> >
>> > Bertram Ludaescher wrote:
>> >  >> Matt et al:
>> >>
>> >> Can't one get the best of both worlds:
>> >> (1) fix the "~/.kepler bug" to avoid shooting yourself in the foot 
>> with
>> >> multiple running instances
>> >> (2) do data caching
>> >>
>> >> It seems that it's a consequence of the current implementation that
>> >> they are interdependent.
>> >>
>> >> The workaround seems like a good short-term solution (I mean
>> >> workaround). Also, maybe this whole caching thing needs to be
>> >> rethought/re-engineered.
>> >>
>> >> Bertram
>> >>
>> >>    >>>>> On Fri, 11 May 2007 08:27:18 -0700
>> >>>>> Matthew Jones <jones at nceas.ucsb.edu> wrote:
>> >>>>>          >> MJ>
>> >> MJ> We've known about this issue for a while, and recognize that 
>> we need a
>> >> MJ> better solution.
>> >> MJ>
>> >> MJ> The issue is that Kepler uses a backend relational db for 
>> caching and
>> >> MJ> several data processing activities.  That relational db (hsql) 
>> stores
>> >> MJ> its files in a subdir of the ~/.kepler directory.  If more 
>> than one
>> >> MJ> instance of Kepler tries to start up, you get the exception that
>> >> MJ> Christopher listed.
>> >> MJ>
>> >> MJ> One potential workaround is to have the script that runs the 
>> multiple
>> >> MJ> instances run from different accounts, and therefore each 
>> instance
>> >> would
>> >> MJ> use a different .kepler folder and therefore different .hsql 
>> db.  The
>> >> MJ> limitation is not really on running more than one instance of
>> >> kepler on
>> >> MJ> a machine -- its running more than one instance of Kepler in a 
>> single
>> >> MJ> user account.  Any workaround that allows each process to have 
>> its own
>> >> MJ> .kepler dir will work.  Of course, the downside is that these
>> >> processes
>> >> MJ> will not share the cache, and therefore large data downloads, 
>> etc,
>> >> will
>> >> MJ> have to be repeated for each instance.
>> >> MJ>
>> >> MJ> Matt
>> >> MJ>
>> >> MJ> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> >> MJ> Matthew B. Jones
>> >> MJ> jones at nceas.ucsb.edu                Ph: 541-888-2581 ext. 287
>> >> MJ> National Center for Ecological Analysis and Synthesis (NCEAS)
>> >> MJ> UC Santa Barbara
>> >> MJ> http://www.nceas.ucsb.edu/ecoinformatics
>> >> MJ> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> >> MJ>
>> >> MJ>
>> >> MJ> Christopher Brooks wrote:
>> >>    >>>> Hi Norbert,
>> >>>>
>> >>>> Your solution will probably run Ptolemy models that do not use 
>> the UI.
>> >>>> I'm not sure if it will run Kepler models that do anything fancy 
>> with
>> >>>> ontologies etc.
>> >>>>
>> >>>> Also, I had to set the classpath.
>> >>>> I got:
>> >>>> --start--
>> >>>> #! /bin/bash
>> >>>> java -Xmx512m -Xss5m -DKEPLER="$KEPLER" \
>> >>>> -classpath "${PTII};${KEPLER}" \
>> >>>> -Djava.endorsed.dirs=$KEPLER/lib/jar/apache \
>> >>>> -Djava.library.path=$KEPLER/lib
>> >>>> ptolemy.actor.gui.MoMLSimpleApplication $*
>> >>>> --end--
>> >>>>
>> >>>> For info about invoking Ptolemy II, see
>> >>>>
>> >>>>        >> 
>> http://ptolemy.berkeley.edu/ptolemyII/ptIIfaq.htm#invoking%20Ptolemy%20II? 
>>
>> >>    >>>> However, invoking Kepler twice on the same machine via 
>> "ant run-dev"
>> >>>> does not work.  I get:
>> >>>>
>> >>>> [java] Caused by: java.sql.SQLException: The database is already
>> >>>> in use by another process: org.hsqldb.NIOLockFile at f0f563da[file
>> >>>>        >> =C:\Documents
>> >>    >>>> and Settings\cxh\.kepler\cache\cachedata\hsqldb.lck, 
>> exists=true,
>> >>>>        >> locked=false,
>> >>    >>>> valid=false,fl =null]: java.lang.Exception: The process 
>> cannot
>> >>>>        >> access the file
>> >>    >>>> because another process has locked a portion of the file :
>> >>>> C:\Documents and Settings\cxh\.kepler\cache\cachedata\hsqldb.lck
>> >>>> [java]     at org.hsqldb.jdbc.jdbcUtil.sqlException(Unknown Source)
>> >>>> [java]     at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
>> >>>> [java]     at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
>> >>>> [java]     at org.hsqldb.jdbcDriver.connect(Unknown Source)
>> >>>> [java]     at
>> >>>>        >> 
>> java.sql.DriverManager.getConnection(DriverManager.java:525)
>> >>    >>>> [java]     at
>> >>>>        >> 
>> java.sql.DriverManager.getConnection(DriverManager.java:171)
>> >>    >>>> [java]     at
>> >>>>        >> 
>> org.ecoinformatics.util.DBConnectionFactory.getDBConnection(DBConnectionFactory.java:91) 
>>
>> >>    >>>> [java]     at
>> >>>>        >> 
>> org.ecoinformatics.util.DBConnectionFactory.getDBConnection(DBConnectionFactory.java:73) 
>>
>> >>    >>>> [java]     at
>> >>>>        >> 
>> org.kepler.gui.KeplerInitializer.initializeSystem(KeplerInitializer.java:133) 
>>
>> >>    >>>> [java]     at
>> >>>>        >> 
>> org.kepler.gui.KeplerInitializer.<init>(KeplerInitializer.java:80)
>> >>    >>>> [java]     at
>> >>>>        >> 
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> >>    >>>> [java]     at
>> >>>>        >> 
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) 
>>
>> >>    >>>> [java]     at
>> >>>>        >> 
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) 
>>
>> >>    >>>> [java]     at
>> >>>>        >> 
>> java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>> >>    >>>> [java]     at java.lang.Class.newInstance0(Class.java:350)
>> >>>> [java]     at java.lang.Class.newInstance(Class.java:303)
>> >>>> [java]     at
>> >>>>        >> 
>> ptolemy.actor.gui.MoMLApplication.readConfiguration(MoMLApplication.java:396) 
>>
>> >>    >>>> [java]     ... 14 more
>> >>>>
>> >>>> BUILD SUCCESSFUL
>> >>>> Total time: 9 seconds
>> >>>>
>> >>>>
>> >>>>
>> >>>> See
>> >>>> http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2315
>> >>>> This bug was marked as WONTFIX by Chad, who wrote:
>> >>>>
>> >>>>        >>>>> The only way we could fix this is to have multiple 
>> .kepler
>> >>>>> directories on one machine. This would require a profiling system
>> >>>>> like morpho has.  Since I don't think this is on our list of 
>> things
>> >>>>> to do, i'm going to mark this WONTFIX.  If at some point we add a
>> >>>>> profile system, possibly when we add authentication, this problem
>> >>>>> should be fixed.
>> >>>>>          >>>> _Christopher
>> >>>>
>> >>>>
>> >>>> --------
>> >>>>
>> >>>> Hi Lisong,
>> >>>>
>> >>>> If you want to execute workflows from command-line, just create a
>> >>>>        >> script
>> >>    >>>> similar to kepler.sh and replace the java command for the 
>> following:
>> >>>>
>> >>>> java -Xmx512m -Xss5m -DKEPLER="$KEPLER"
>> >>>> -Djava.endorsed.dirs=$KEPLER/lib/jar/apache
>> >>>> -Djava.library.path=$KEPLER/lib
>> >>>>        >> ptolemy.actor.gui.MoMLSimpleApplication $*
>> >>    >>>> (note, KEPLER should be defined for the above script)
>> >>>>
>> >>>> This can be executed several times at once, giving the workflow 
>> name as
>> >>>> parameter.
>> >>>>
>> >>>> Best regards
>> >>>> Norbert
>> >>>>
>> >>>> Norbert Podhorszki
>> >>>> ------------------------------------
>> >>>> University of California, Davis
>> >>>> Department of Computer Science
>> >>>> 1 Shields Ave, 2236 Kemper Hall
>> >>>> Davis, CA 95616
>> >>>> (530) 752-5076
>> >>>> pnorbert at cs.ucdavis.edu
>> >>>> ----------------------------------
>> >>>>
>> >>>> On Thu, 10 May 2007, Sun, Lisong wrote:
>> >>>>
>> >>>>        >>>>> Dear Kepler developers,
>> >>>>>
>> >>>>> Our group is developing a process integration framework which use
>> >>>>>          >> Kepler
>> >>    >>>> to
>> >>>>        >>>>> build and run workflows to do scientific 
>> computations. We built
>> >>>>>          >> an organi
>> >>    >>>> zer
>> >>>>        >>>>> application that will manage the workflow files on 
>> the WebDAV server,
>> >>>>> input/output for each run and other provenance information.
>> >>>>>
>> >>>>> So far we are very impressed by the capabilities of Kepler.
>> >>>>>          >> Meanwhile we
>> >>    >>>> do
>> >>>>        >>>>> run into a few problems that we are not sure how to 
>> deal with.
>> >>>>>
>> >>>>> One of the problem is that we found Kepler can only be invoked
>> >>>>>          >> once. And
>> >>    >>>> if
>> >>>>        >>>>> there is a running Kepler, the second invoke 
>> command will fail.
>> >>>>>          >> This does
>> >>    >>>>> get in our way of using the Kepler since we manage all the
>> >>>>>          >> workflow files
>> >>    >>>>> using our organizer and need to be able to invoke multiple
>> >>>>>          >> workflows thro
>> >>    >>>> ugh
>> >>>>        >>>>> the organizer. We do noticed that it is possible to 
>> open multiple
>> >>>>>          >> workflo
>> >>    >>>> ws
>> >>>>        >>>>> windows through the menu options. Wonder if there 
>> is any special
>> >>>>>          >> reason w
>> >>    >>>> hy
>> >>>>        >>>>> it can't run with second invoke command.
>> >>>>>
>> >>>>> Another goal we want to achieve is to deal with long running
>> >>>>>          >> workflows. I
>> >>    >>>>> wonder if there is anyway we can use the Kepler as an 
>> execution
>> >>>>>          >> engine an
>> >>    >>>> d
>> >>>>        >>>>> run workflows without open the GUI window.
>> >>>>>
>> >>>>> Thanks for your help.
>> >>>>>
>> >>>>> Best regards,
>> >>>>>
>> >>>>> Lisong Sun
>> >>>>> Staff Scientist
>> >>>>> Pacific Northwest National Laboratory
>> >>>>> _______________________________________________
>> >>>>> Kepler-dev mailing list
>> >>>>> Kepler-dev at ecoinformatics.org
>> >>>>>
>> >>>>>          >> 
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>> >>    >>>> _______________________________________________
>> >>>> Kepler-dev mailing list
>> >>>> Kepler-dev at ecoinformatics.org
>> >>>>
>> >>>>        >> 
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>> >>    >>>> --------
>> >>>> _______________________________________________
>> >>>> Kepler-dev mailing list
>> >>>> Kepler-dev at ecoinformatics.org
>> >>>>
>> >>>>        >> 
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>> >> MJ> _______________________________________________
>> >> MJ> Kepler-dev mailing list
>> >> MJ> Kepler-dev at ecoinformatics.org
>> >> MJ>
>> >> 
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>> >>
>> >> _______________________________________________
>> >> Kepler-dev mailing list
>> >> Kepler-dev at ecoinformatics.org
>> >> 
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>> >>
>> >>    > _______________________________________________
>> > Kepler-dev mailing list
>> > Kepler-dev at ecoinformatics.org
>> > 
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>> > 
>>
>> -- 
>> *******************************************************************
>> Dan Higgins                                  higgins at nceas.ucsb.edu
>> http://www.nceas.ucsb.edu/    Ph: 805-893-5127
>> National Center for Ecological Analysis and Synthesis (NCEAS) Marine 
>> Science Building - Room 3405
>> Santa Barbara, CA 93195
>> *******************************************************************
>>
>



More information about the Kepler-dev mailing list