[kepler-dev] Q: SGE command execution on remote SGE cluster started on local PC fails due to environment problems

[Jianwu Wang]

Hi Johann,

     Can you try adding SGE_CELL and SGE_ROOT settings at 'job submit 
options' parameter of GenericJobLauncher actor or 'jobSubmitOptions' 
parameter of JobSubmitter actor? An example is 
"SGE_CELL=hoffman2,SGE_ROOT=/u/systems/SGE6.1u3". You might also need to 
set the path for your qsub (such as 
"/u/systems/SGE6.2u4/bin/lx26-amd64") to the 'binary path' parameter of 
GenericJobLauncher actor or 'binPath' parameter of JobManager actor. I 
remember I met a similar problem before and this did the trick.

Best wishes

Sincerely yours

Jianwu WANG, Ph.D.
jianwu at sdsc.edu
http://users.sdsc.edu/~jianwu/

Assistant Director for Research
Workflows for Data Science (WorDS) Center of Excellence
San Diego Supercomputer Center (SDSC)
University of California, San Diego (UCSD)

On 2/6/15 3:31 PM, Hoeftberger, Johann wrote:
> Hello,
>
> I try to create my own simple Kepler test workflow on my local PC and
> run it via SSH on a SGE cluster.
> For that I took the Kepler demo Workflow
> "Job_Submission_Using_JobManager" configured it for my local situation
> an tried to run it.
> I know that the connection to the cluster, the login with the given
> credentials and the settings for the working directory work well. (I get
> created directories and files for the cluster jobs which should be
> executed.)
>
> The execution of the qsub command on the cluster doesn't work because at
> first I got the exception "qsub: unknown command" and when I hardcoded
> the full path for the qsub command in the implementation (to locate the
> error only), I changed the initialization of private String
> _sgeSubmitCmd in JobSupportSGE.java to the full path of the qsub command
> on my SGE cluster, I got the next exception "Unable to initialize
> environment because of error: Please set the environment variable SGE_ROOT".
>
> SGE_ROOT is properly set on the SGE cluster. I tried to set it
> additionally on my local PC (afterwards I restarted Eclipse where my
> Kepler instance is running) and in the Eclipse - Run - Run Configuration
> - Java Application - Environment. All these tries didn't solve the
> problem, I still get the same exception about the uninitialized environment.
>
> I found in SshExec.java::public int executeCmd(String command,
> OutputStream streamOut, OutputStream streamErr, String thirdPartyTarget)
> the documentation
>
> /**
> 	 * Execute a command on the remote machine and expect a password/passphrase
> 	 * question from the command. The stream <i>streamOut</i> should be
> provided
> 	 * to get the output and errors merged. <i>streamErr</i> is not used in
> this
> 	 * method (it will be empty string finally).
> 	 *
> 	 * @return exit code of command if execution succeeded,
> 	 * @throws ExecTimeoutException
> 	 *             if the command failed because of timeout
> 	 * @throws SshException
> 	 *             if an error occurs for the ssh connection during the command
> 	 *             execution Note: in this method, the SSH Channel is forcing a
> 	 *             pseudo-terminal allocation {see setPty(true)} to allow
> remote
> 	 *             commands to read something from their stdin (i.e. from us
> 	 *             here), thus, (1) remote environment is not set from
> 	 *             .bashrc/.cshrc and (2) stdout and stderr come back merged in
> 	 *             one stream.
> 	 */
>
> so I guess my problem is caused through an uninitialized or wrong
> initialized used (pseudo) terminal.
> For me it seems the set environment variables on the used systems (SGE
> cluster, local PC) are not read and the Kepler implementation itself
> doesn't set proper values for the needed variables.
>
> I haven't found a description or a configuration possibility for my
> issue yet. So I think it is some kind of implementation flaw.
>
> Can somebody give me a hint how to solve this issue. I would like to run
> Kepler locally on my PC but execute parts of my Workflow remotely on my
> SGE cluster.
>
>
> Kind regards,
> Johann Hoeftberger
>
>