[kepler-dev] kepler start failure on concurrent runs

Daniel Crawl danielcrawl at gmail.com
Wed Nov 28 11:51:23 PST 2012 

Hi Emanuel,

The first Kepler instance starts an HSQL server, and subsequent Kepler
instances connect to this server. When the first instance exits, the
HSQL server is stopped and all its connections are closed. If you want
to run multiple instances from the command line, you can start the HSQL
server separately using "-hsql start":

kepler.sh -hsql start
kepler.sh -runkar workflow1.kar
kepler.sh -runkar workflow2.kar
...
kepler.sh -hsql stop

I just fixed a bug related to starting the HSQL server, so you'll need
to update your sources.

   --dan

On 11/28/12 6:25 AM, Emanuel Dima wrote:
> Hi,
>
> I was testing kepler performance for a specific use case by
> concurrently starting multiple instances and running the same .kar
> workflow. I noticed that from time to time kepler refuses to start and
> prints an exception stack.
>
> After digging through the available docs I hypothesized that this must
> be caused by the kepler cache being used concurrently and that the
> -nocache option should solve the problem, which it did (I also had to
> change the workflow file format from .kar to .xml).
>
> The question I have is if this behavior is considered normal or is a
> bug that I should report.
>
> For the record, an example of exception log is:
>
> ssh -p 9875 root at localhost time
> /data/install-kepler/kepler-2.3/kepler.sh -runwf -nogui
> /data/WorkflowEngine/engine/kepler/execute_external_sleep_command.kar
> The base dir is /data/install-kepler/kepler-2.3
> Kepler.run going to run.setMain(org.kepler.Kepler)
> JVM Memory = 5m 512m
> RunClasspath:kepler-2.3.0 /data/install-kepler/kepler-2.3/kepler-2.3.0/src
> RunClasspath:outreach-2.2.0 /data/install-kepler/kepler-2.3/outreach-2.2.0/src
> RunClasspath:r-2.3.0 /data/install-kepler/kepler-2.3/r-2.3.0/src
> RunClasspath:loader-2.2.0 /data/install-kepler/kepler-2.3/loader-2.2.0/src
> RunClasspath:actors-2.3.0 /data/install-kepler/kepler-2.3/actors-2.3.0/src
> RunClasspath:directors-2.0.0 /data/install-kepler/kepler-2.3/directors-2.0.0/src
> RunClasspath:opendap-2.0.0 /data/install-kepler/kepler-2.3/opendap-2.0.0/src
> RunClasspath:dataturbine-2.2.0
> /data/install-kepler/kepler-2.3/dataturbine-2.2.0/src
> RunClasspath:ecogrid-2.3.0 /data/install-kepler/kepler-2.3/ecogrid-2.3.0/src
> RunClasspath:authentication-gui-2.2.0
> /data/install-kepler/kepler-2.3/authentication-gui-2.2.0/src
> RunClasspath:gui-2.3.0 /data/install-kepler/kepler-2.3/gui-2.3.0/src
> RunClasspath:module-manager-gui-2.3.0
> /data/install-kepler/kepler-2.3/module-manager-gui-2.3.0/src
> RunClasspath:authentication-2.3.0
> /data/install-kepler/kepler-2.3/authentication-2.3.0/src
> RunClasspath:repository-2.3.0
> /data/install-kepler/kepler-2.3/repository-2.3.0/src
> RunClasspath:job-2.1.0 /data/install-kepler/kepler-2.3/job-2.1.0/src
> RunClasspath:io-2.0.0 /data/install-kepler/kepler-2.3/io-2.0.0/src
> RunClasspath:ssh-2.1.0 /data/install-kepler/kepler-2.3/ssh-2.1.0/src
> RunClasspath:data-handling-2.1.0
> /data/install-kepler/kepler-2.3/data-handling-2.1.0/src
> RunClasspath:sms-2.2.0 /data/install-kepler/kepler-2.3/sms-2.2.0/src
> RunClasspath:component-library-2.2.0
> /data/install-kepler/kepler-2.3/component-library-2.2.0/src
> RunClasspath:util-2.1.0 /data/install-kepler/kepler-2.3/util-2.1.0/src
> RunClasspath:event-state-2.1.0
> /data/install-kepler/kepler-2.3/event-state-2.1.0/src
> RunClasspath:core-2.3.0 /data/install-kepler/kepler-2.3/core-2.3.0/src
> RunClasspath:common-2.3.0 /data/install-kepler/kepler-2.3/common-2.3.0/src
> RunClasspath:module-manager-2.3.0
> /data/install-kepler/kepler-2.3/module-manager-2.3.0/src
> RunClasspath:configuration-manager-2.3.0
> /data/install-kepler/kepler-2.3/configuration-manager-2.3.0/src
> RunClasspath:kepler-tasks-2.3.0
> /data/install-kepler/kepler-2.3/kepler-tasks-2.3.0/src
> RunClasspath:ptolemy-kepler-2.3.0
> /data/install-kepler/kepler-2.3/ptolemy-kepler-2.3.0/src
>       [null] Checking for patches...
>       [null] Ran additional initialization for module core from class
> org.kepler.module.core.Initialize
>       [null] Ran additional initialization for module gui from class
> org.kepler.module.gui.Initialize
>       [null] loading: org.kepler.KeplerConfigurationApplication args:
> ptolemy/configs/kepler/ConfigNoGUIWithCache.xml -runThenExit
> /data/WorkflowEngine/engine/kepler/execute_external_sleep_command.kar
>       [null] invoking: org.kepler.KeplerConfigurationApplication.main
>       [null] Kepler Initializing...
>       [null] Unsupported file type or connection not available:
> file:/data/WorkflowEngine/engine/kepler/execute_external_sleep_command.kar
>       [null] Extracting workflow from KAR
>       [null] WARN  (org.kepler.kar.KARFile:cacheKARContents:573) KAR
> should be in a Local Repository Folder to be inserted in the cache:
> /data/WorkflowEngine/engine/kepler/execute_external_sleep_command.kar
>       [null] Cached KAR contents
>       [null] Processing entry,
> LSID=urn:lsid:kepler-project.org/ns/:17606:39:111,
> type=ptolemy.actor.TypedCompositeActor
>       [null] Running workflow /tmp/moml6081054809666236546.xml
> extracted from kar file
> /data/WorkflowEngine/engine/kepler/execute_external_sleep_command.kar
>       [null] 1053 ms. Memory: 89664K Free: 26607K (30%)
>       [null] Ran additional cleanup for module core from class
> org.kepler.module.core.Shutdown
> it>(KeplerConfigurationApplication.java:191)
>       [null] at org.kepler.KeplerConfigurationApplication.main(KeplerConfigurationApplication.java:639)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>       [null] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>       [null] at java.lang.reflect.Method.invoke(Method.java:597)
>       [null] at org.kepler.Kepler.load(Kepler.java:440)
>       [null] at org.kepler.Kepler.load(Kepler.java:465)
>       [null] at org.kepler.Kepler.parseArgsAndRun(Kepler.java:379)
>       [null] at org.kepler.Kepler.main(Kepler.java:260)
>       [null] Cached KAR contents
>       [null] Processing entry,
> LSID=urn:lsid:kepler-project.org/ns/:17606:39:111,
> type=ptolemy.actor.TypedCompositeActor
>       [null] java.sql.SQLException: Connection is broken: java.io.EOFException
>       [null] at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
>       [null] at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source)
>       [null] at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source)
>       [null] at org.kepler.objectmanager.cache.CacheManager.getObject(CacheManager.java:508)
>       [null] at org.kepler.KeplerConfigurationApplication.<init>(KeplerConfigurationApplication.java:207)
>       [null] at org.kepler.KeplerConfigurationApplication.main(KeplerConfigurationApplication.java:639)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>       [null] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>       [null] at java.lang.reflect.Method.invoke(Method.java:597)
>       [null] at org.kepler.Kepler.load(Kepler.java:440)
>       [null] at org.kepler.Kepler.load(Kepler.java:465)
>       [null] at org.kepler.Kepler.parseArgsAndRun(Kepler.java:379)
>       [null] at org.kepler.Kepler.main(Kepler.java:260)
>       [null] Command failed
>       [null] java.lang.Exception: Failed to parse
> "ptolemy/configs/kepler/ConfigNoGUIWithCache.xml -runThenExit
> /data/WorkflowEngine/engine/kepler/execute_external_sleep_command.kar"
>       [null] at ptolemy.actor.gui.ConfigurationApplication.throwArgsException(ConfigurationApplication.java:791)
>       [null] at org.kepler.KeplerConfigurationApplication.<init>(KeplerConfigurationApplication.java:339)
>       [null] at org.kepler.KeplerConfigurationApplication.main(KeplerConfigurationApplication.java:639)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>       [null] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>       [null] at java.lang.reflect.Method.invoke(Method.java:597)
>       [null] at org.kepler.Kepler.load(Kepler.java:440)
>       [null] at org.kepler.Kepler.load(Kepler.java:465)
>       [null] at org.kepler.Kepler.parseArgsAndRun(Kepler.java:379)
>       [null] at org.kepler.Kepler.main(Kepler.java:260)
>       [null] Caused by: org.kepler.objectmanager.cache.CacheException:
> SQL exception when getting object
>       [null] at org.kepler.objectmanager.cache.CacheManager.getObject(CacheManager.java:533)
>       [null] at org.kepler.KeplerConfigurationApplication.<init>(KeplerConfigurationApplication.java:207)
>       [null] ... 9 more
>       [null] Caused by: java.sql.SQLException: Connection is broken:
> java.io.EOFException
>       [null] at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
>       [null] at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source)
>       [null] at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source)
>       [null] at org.kepler.objectmanager.cache.CacheManager.getObject(CacheManager.java:508)
>       [null] ... 10 more
>       [null] java.lang.Exception: Failed to parse
> "ptolemy/configs/kepler/ConfigNoGUIWithCache.xml -runThenExit
> /data/WorkflowEngine/engine/kepler/execute_external_sleep_command.kar"
>       [null] at ptolemy.actor.gui.ConfigurationApplication.throwArgsException(ConfigurationApplication.java:791)
>       [null] at org.kepler.KeplerConfigurationApplication.<init>(KeplerConfigurationApplication.java:339)
>       [null] at org.kepler.KeplerConfigurationApplication.main(KeplerConfigurationApplication.java:639)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       [null] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>       [null] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>       [null] at java.lang.reflect.Method.invoke(Method.java:597)
>       [null] at org.kepler.Kepler.load(Kepler.java:440)
>       [null] at org.kepler.Kepler.load(Kepler.java:465)
>       [null] at org.kepler.Kepler.parseArgsAndRun(Kepler.java:379)
>       [null] at org.kepler.Kepler.main(Kepler.java:260)
>       [null] Caused by: org.kepler.objectmanager.cache.CacheException:
> SQL exception when getting object
>       [null] at org.kepler.objectmanager.cache.CacheManager.getObject(CacheManager.java:533)
>       [null] at org.kepler.KeplerConfigurationApplication.<init>(KeplerConfigurationApplication.java:207)
>       [null] ... 9 more
>       [null] Caused by: java.sql.SQLException: Connection is broken:
> java.io.EOFException
>       [null] at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
>       [null] at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source)
>       [null] at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source)
>       [null] at org.kepler.objectmanager.cache.CacheManager.getObject(CacheManager.java:508)
>       [null] ... 10 more
>
> Thanks,
> Emanuel Dima
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at kepler-project.org
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>

More information about the Kepler-dev mailing list