[kepler-dev] Kepler can only be invoked once
Dan Higgins
higgins at nceas.ucsb.edu
Tue May 15 13:40:26 PDT 2007
Hi Lisong,
One way of getting around the data cache problem is to use
'momlexecute' to run kepler workflows. I have put together a new page on
the Kepler wiki describing different ways of launching Kepler from the
command line - see
http://www.kepler-project.org/Wiki.jsp?page=ExecutingKepler
Dan Higgins - NCEAS
Lisong Sun wrote:
> Thank your all for the helpful info. About the multiple running problem,
> we probably won't care much about data caching. So what does it take in
> order to make it working. For example, if the data cache directory is
> controlled by an environment variable, we can generate unique temp dir
> for each running instance and pass in the path for kepler to use. Or
> simply turn off the data caching totally if that is possible.
>
> Lisong
>
> Bertram Ludaescher wrote:
>
>> Matt et al:
>>
>> Can't one get the best of both worlds:
>> (1) fix the "~/.kepler bug" to avoid shooting yourself in the foot with
>> multiple running instances
>> (2) do data caching
>>
>> It seems that it's a consequence of the current implementation that
>> they are interdependent.
>>
>> The workaround seems like a good short-term solution (I mean
>> workaround). Also, maybe this whole caching thing needs to be
>> rethought/re-engineered.
>>
>> Bertram
>>
>>
>>>>> On Fri, 11 May 2007 08:27:18 -0700
>>>>> Matthew Jones <jones at nceas.ucsb.edu> wrote:
>>>>>
>> MJ>
>> MJ> We've known about this issue for a while, and recognize that we need a
>> MJ> better solution.
>> MJ>
>> MJ> The issue is that Kepler uses a backend relational db for caching and
>> MJ> several data processing activities. That relational db (hsql) stores
>> MJ> its files in a subdir of the ~/.kepler directory. If more than one
>> MJ> instance of Kepler tries to start up, you get the exception that
>> MJ> Christopher listed.
>> MJ>
>> MJ> One potential workaround is to have the script that runs the multiple
>> MJ> instances run from different accounts, and therefore each instance
>> would
>> MJ> use a different .kepler folder and therefore different .hsql db. The
>> MJ> limitation is not really on running more than one instance of
>> kepler on
>> MJ> a machine -- its running more than one instance of Kepler in a single
>> MJ> user account. Any workaround that allows each process to have its own
>> MJ> .kepler dir will work. Of course, the downside is that these
>> processes
>> MJ> will not share the cache, and therefore large data downloads, etc,
>> will
>> MJ> have to be repeated for each instance.
>> MJ>
>> MJ> Matt
>> MJ>
>> MJ> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> MJ> Matthew B. Jones
>> MJ> jones at nceas.ucsb.edu Ph: 541-888-2581 ext. 287
>> MJ> National Center for Ecological Analysis and Synthesis (NCEAS)
>> MJ> UC Santa Barbara
>> MJ> http://www.nceas.ucsb.edu/ecoinformatics
>> MJ> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> MJ>
>> MJ>
>> MJ> Christopher Brooks wrote:
>>
>>>> Hi Norbert,
>>>>
>>>> Your solution will probably run Ptolemy models that do not use the UI.
>>>> I'm not sure if it will run Kepler models that do anything fancy with
>>>> ontologies etc.
>>>>
>>>> Also, I had to set the classpath.
>>>> I got:
>>>> --start--
>>>> #! /bin/bash
>>>> java -Xmx512m -Xss5m -DKEPLER="$KEPLER" \
>>>> -classpath "${PTII};${KEPLER}" \
>>>> -Djava.endorsed.dirs=$KEPLER/lib/jar/apache \
>>>> -Djava.library.path=$KEPLER/lib
>>>> ptolemy.actor.gui.MoMLSimpleApplication $*
>>>> --end--
>>>>
>>>> For info about invoking Ptolemy II, see
>>>>
>>>>
>> http://ptolemy.berkeley.edu/ptolemyII/ptIIfaq.htm#invoking%20Ptolemy%20II?
>>
>>>> However, invoking Kepler twice on the same machine via "ant run-dev"
>>>> does not work. I get:
>>>>
>>>> [java] Caused by: java.sql.SQLException: The database is already
>>>> in use by another process: org.hsqldb.NIOLockFile at f0f563da[file
>>>>
>> =C:\Documents
>>
>>>> and Settings\cxh\.kepler\cache\cachedata\hsqldb.lck, exists=true,
>>>>
>> locked=false,
>>
>>>> valid=false,fl =null]: java.lang.Exception: The process cannot
>>>>
>> access the file
>>
>>>> because another process has locked a portion of the file :
>>>> C:\Documents and Settings\cxh\.kepler\cache\cachedata\hsqldb.lck
>>>> [java] at org.hsqldb.jdbc.jdbcUtil.sqlException(Unknown Source)
>>>> [java] at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
>>>> [java] at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
>>>> [java] at org.hsqldb.jdbcDriver.connect(Unknown Source)
>>>> [java] at
>>>>
>> java.sql.DriverManager.getConnection(DriverManager.java:525)
>>
>>>> [java] at
>>>>
>> java.sql.DriverManager.getConnection(DriverManager.java:171)
>>
>>>> [java] at
>>>>
>> org.ecoinformatics.util.DBConnectionFactory.getDBConnection(DBConnectionFactory.java:91)
>>
>>>> [java] at
>>>>
>> org.ecoinformatics.util.DBConnectionFactory.getDBConnection(DBConnectionFactory.java:73)
>>
>>>> [java] at
>>>>
>> org.kepler.gui.KeplerInitializer.initializeSystem(KeplerInitializer.java:133)
>>
>>>> [java] at
>>>>
>> org.kepler.gui.KeplerInitializer.<init>(KeplerInitializer.java:80)
>>
>>>> [java] at
>>>>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>
>>>> [java] at
>>>>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>
>>>> [java] at
>>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>
>>>> [java] at
>>>>
>> java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>>
>>>> [java] at java.lang.Class.newInstance0(Class.java:350)
>>>> [java] at java.lang.Class.newInstance(Class.java:303)
>>>> [java] at
>>>>
>> ptolemy.actor.gui.MoMLApplication.readConfiguration(MoMLApplication.java:396)
>>
>>>> [java] ... 14 more
>>>>
>>>> BUILD SUCCESSFUL
>>>> Total time: 9 seconds
>>>>
>>>>
>>>>
>>>> See
>>>> http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2315
>>>> This bug was marked as WONTFIX by Chad, who wrote:
>>>>
>>>>
>>>>> The only way we could fix this is to have multiple .kepler
>>>>> directories on one machine. This would require a profiling system
>>>>> like morpho has. Since I don't think this is on our list of things
>>>>> to do, i'm going to mark this WONTFIX. If at some point we add a
>>>>> profile system, possibly when we add authentication, this problem
>>>>> should be fixed.
>>>>>
>>>> _Christopher
>>>>
>>>>
>>>> --------
>>>>
>>>> Hi Lisong,
>>>>
>>>> If you want to execute workflows from command-line, just create a
>>>>
>> script
>>
>>>> similar to kepler.sh and replace the java command for the following:
>>>>
>>>> java -Xmx512m -Xss5m -DKEPLER="$KEPLER"
>>>> -Djava.endorsed.dirs=$KEPLER/lib/jar/apache
>>>> -Djava.library.path=$KEPLER/lib
>>>>
>> ptolemy.actor.gui.MoMLSimpleApplication $*
>>
>>>> (note, KEPLER should be defined for the above script)
>>>>
>>>> This can be executed several times at once, giving the workflow name as
>>>> parameter.
>>>>
>>>> Best regards
>>>> Norbert
>>>>
>>>> Norbert Podhorszki
>>>> ------------------------------------
>>>> University of California, Davis
>>>> Department of Computer Science
>>>> 1 Shields Ave, 2236 Kemper Hall
>>>> Davis, CA 95616
>>>> (530) 752-5076
>>>> pnorbert at cs.ucdavis.edu
>>>> ----------------------------------
>>>>
>>>> On Thu, 10 May 2007, Sun, Lisong wrote:
>>>>
>>>>
>>>>> Dear Kepler developers,
>>>>>
>>>>> Our group is developing a process integration framework which use
>>>>>
>> Kepler
>>
>>>> to
>>>>
>>>>> build and run workflows to do scientific computations. We built
>>>>>
>> an organi
>>
>>>> zer
>>>>
>>>>> application that will manage the workflow files on the WebDAV server,
>>>>> input/output for each run and other provenance information.
>>>>>
>>>>> So far we are very impressed by the capabilities of Kepler.
>>>>>
>> Meanwhile we
>>
>>>> do
>>>>
>>>>> run into a few problems that we are not sure how to deal with.
>>>>>
>>>>> One of the problem is that we found Kepler can only be invoked
>>>>>
>> once. And
>>
>>>> if
>>>>
>>>>> there is a running Kepler, the second invoke command will fail.
>>>>>
>> This does
>>
>>>>> get in our way of using the Kepler since we manage all the
>>>>>
>> workflow files
>>
>>>>> using our organizer and need to be able to invoke multiple
>>>>>
>> workflows thro
>>
>>>> ugh
>>>>
>>>>> the organizer. We do noticed that it is possible to open multiple
>>>>>
>> workflo
>>
>>>> ws
>>>>
>>>>> windows through the menu options. Wonder if there is any special
>>>>>
>> reason w
>>
>>>> hy
>>>>
>>>>> it can't run with second invoke command.
>>>>>
>>>>> Another goal we want to achieve is to deal with long running
>>>>>
>> workflows. I
>>
>>>>> wonder if there is anyway we can use the Kepler as an execution
>>>>>
>> engine an
>>
>>>> d
>>>>
>>>>> run workflows without open the GUI window.
>>>>>
>>>>> Thanks for your help.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Lisong Sun
>>>>> Staff Scientist
>>>>> Pacific Northwest National Laboratory
>>>>> _______________________________________________
>>>>> Kepler-dev mailing list
>>>>> Kepler-dev at ecoinformatics.org
>>>>>
>>>>>
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>>
>>>> _______________________________________________
>>>> Kepler-dev mailing list
>>>> Kepler-dev at ecoinformatics.org
>>>>
>>>>
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>>
>>>> --------
>>>> _______________________________________________
>>>> Kepler-dev mailing list
>>>> Kepler-dev at ecoinformatics.org
>>>>
>>>>
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>> MJ> _______________________________________________
>> MJ> Kepler-dev mailing list
>> MJ> Kepler-dev at ecoinformatics.org
>> MJ>
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>>
>> _______________________________________________
>> Kepler-dev mailing list
>> Kepler-dev at ecoinformatics.org
>> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>>
>>
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at ecoinformatics.org
> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>
--
*******************************************************************
Dan Higgins higgins at nceas.ucsb.edu
http://www.nceas.ucsb.edu/ Ph: 805-893-5127
National Center for Ecological Analysis and Synthesis (NCEAS) Marine Science Building - Room 3405
Santa Barbara, CA 93195
*******************************************************************
More information about the Kepler-dev
mailing list