[kepler-users] Sample workflow using JobSubmission/JobManager

Tue Nov 6 14:33:15 PST 2012

Thank you both for the explanation. I will engage the Jsch folks when the
time comes to do this work. Conversely I wonder what other ssh java
libraries may support this mode, over which a new actor can be built.

On 11/6/12 10:16 AM, "Podhorszki, Norbert" <pnorbert at ornl.gov> wrote:

>Hi,
>
>Kepler is using JSch, which is a java implementation of the client side
>of the ssh protocol, and as such, it can utilize nothing from other ssh
>client libraries. The org.keplerSshSession class over JSch.Session is
>basically what a control master does in C client library: it opens and
>authenticates a "session" and then all subsequent operations in the
>SshExec class opens a new "channel" to do a remote operations. It works
>for one step authentications: password, public-key, interactive and GSI
>grid certificates.
>
>What is missing entirely, as Jiangwu explained, that the Kepler classes
>are not prepared for two-stage authentications, so it just cannot work in
>your environment. Someone working in such an environment has to bite the
>bullet and modify  org.kepler.ssh for it.
>
>I have communicated with the JSch authors in the past and they were very
>helpful. Maybe he could help with this too.
>
>Best regards
>Norbert
>
>
>Norbert Podhorszki
> ------------------------------------------
> Scientific Data Group
> Computer Science and Mathematics Division
> Oak Ridge National Laboratory
> Phone: (865) 574-7159
>
>
>
>On Nov 5, 2012, at 8:11 PM, Jianwu Wang wrote:
>
>Hi Eduardo,
>
>    I agree with you. To make it working, Kepler either need to share in
>the externally established SSH connection or establish the master
>connection by itself. Because of the limitation of the underlying JSCH
>library, I don't think SSH Session actor support the first option. The
>second option has to answer two passwords prompted by different screen
>questions. The current SSH Session actor only popup a dialogue for user
>password. I think Kepler code can be extended to support two password
>interaction. Related classes are org.kepler.ssh.SshSession (especially
>MyUserInfo inner class).
>
>    But I don't have access to resources that need two password
>interaction. So it's hard for me to update the code. Are you interested
>in getting your hands dirty on updating Kepler code? :)
>
>Best wishes
>
>Sincerely yours
>
>Jianwu Wang, Ph.D.
>jianwu at sdsc.edu<mailto:jianwu at sdsc.edu>
>http://users.sdsc.edu/~jianwu/
>
>Assistant Project Scientist
>Scientific Workflow Automation Technologies (SWAT) Laboratory
>San Diego Supercomputer Center
>University of California, San Diego
>San Diego, CA, U.S.A.
>
>On 11/5/12 2:18 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE &
>TECHNOLOGY INC] wrote:
>Hi Jianwu,
>
>  The ssh connection is first set up by answering an RSA token password
>followed by a system password.  The RSA token based password changes
>every few seconds.  Since there are many steps (commands) to the remote
>workflow execution we can't have every connection prompt for these
>dynamic passwords.  Therefore it must use the preexisting/authenticated
>connection.  With ssh the first connection becomes the master and
>subsequent ones go straight to the prompt.  You can try that yourself.
>But with Kepler it would have to either establish the master, or use the
>existing system one.  If it were to use the system connection there is no
>password to provide.  If it were to establish the master connection it
>would have to answer two passwords prompted by different screen questions.
>  The bottom line is that Kepler does not seem to share in the externally
>established SSH connection.  I will have to research the JSCH library for
>more information on this mode of operation as well.
>
>Thanks,
>Eduardo
>
>From: Jianwu Wang <jianwu at sdsc.edu<mailto:jianwu at sdsc.edu>>
>Date: Thursday, November 1, 2012 8:12 PM
>To: "Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE & TECHNOLOGY INC]"
><eduardo.g.valente at nasa.gov<mailto:eduardo.g.valente at nasa.gov>>
>Cc: Norbert Podhorszki <pnorbert at ornl.gov<mailto:pnorbert at ornl.gov>>,
>"kepler-users at kepler-project.org<mailto:kepler-users at kepler-project.org>"
><kepler-users at kepler-project.org<mailto:kepler-users at kepler-project.org>>
>Subject: Re: [kepler-users] Sample workflow using JobSubmission/JobManager
>
>Hi Eduardo,
>
>    If you start kepler from kepler.sh/kepler.bat, not double-clicking
>kepler icon, a console will show up and it normally has more information
>on errors.
>
>    I checked the '-M' option of ssh and the implementation of ssh
>related code in Kepler. I don't think Kepler supports control master. I
>also don't see the Jsch library we used for ssh actors support it. So
>it's hard for us to update our code to do it.
>
>    But I think there is still one way workaround. How do you set up the
>ssh connection at the first place? Type your two passwords in command
>line? Does it work if you try it in Kepler using 'SSH Session' actor? The
>same ssh session actor generated by the 'SSH Session' actor can be
>postponed and shared by other workflow executions within the same JVM. To
>do it, the parameter 'postpone' of the actor has to be selected (true)
>and the 'closeAtEnd' has to be false. So if you split the workflow into
>two workflows. The first workflow only has 'Host' and 'SSH Session'. The
>second one has other parts. If you can generate correct ssh session using
>the first workflow. You can run the second workflow many times without
>creating new ssh sessions. My tests work here.
>
>    I'm ccing the email to Norbert. He is the original developer of the
>actors and workflows. He might have better solutions for you.
>
>Best wishes
>
>Sincerely yours
>
>Jianwu Wang, Ph.D.
>jianwu at sdsc.edu<mailto:jianwu at sdsc.edu>http://users.sdsc.edu/~jianwu/<http
>://users.sdsc.edu/%7Ejianwu/>
>
>Assistant Project Scientist
>Scientific Workflow Automation Technologies (SWAT) Laboratory
>San Diego Supercomputer Center
>University of California, San Diego
>San Diego, CA, U.S.A.
>
>On 11/1/12 2:08 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE &
>TECHNOLOGY INC] wrote:
>The failure is graceful in that the session return true for the failed
>port.  If there is a way to view logs for that actor please let me know.
>
>Two factor authentication is the use of two passwords in the admission
>process one of which utilizes an RSA token (changes every 30 seconds or
>so).
>
>This means that it becomes impractical to automate processes if every
>time we ssh the password is different.  And this two factor
>authentication cannot be bypassed with public keys (otherwise known as
>passwordless ssh).  The only option left is to use an existing ssh
>connection enabled as the control master. Generally the M switch of ssh
>clients.  But it would appear the java ssh client and the system ssh
>client do not "see" each other.  If that is the case than the java
>version would need such a mode as well.  Currently I see two modes:
>interactive with password request and passwordless with identity file.
>
>I cannot otherwise provide you a means to recreate the environment we
>have.  But look into the concept of control master and you will be able
>to investigate this possibility with the ssh session actor.
>
>Thanks,
>Eduardo
>
>From: Jianwu Wang <jianwu at sdsc.edu<mailto:jianwu at sdsc.edu>>
>Date: Thursday, November 1, 2012 4:52 PM
>To: "Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE & TECHNOLOGY INC]"
><eduardo.g.valente at nasa.gov<mailto:eduardo.g.valente at nasa.gov>>
>Cc: 
>"kepler-users at kepler-project.org<mailto:kepler-users at kepler-project.org>"
><kepler-users at kepler-project.org<mailto:kepler-users at kepler-project.org>>
>Subject: Re: [kepler-users] Sample workflow using JobSubmission/JobManager
>
>Hi Eduardo,
>
>    So you failed even just using 'SSH Session' actor? Did you get any
>error or exception message? It's new to me for the two-factor
>authentication. If you tell me how to reproduce it, I can dig into it and
>check what went wrong.
>
>Best wishes
>
>Sincerely yours
>
>Jianwu Wang, Ph.D.
>jianwu at sdsc.edu<mailto:jianwu at sdsc.edu>http://users.sdsc.edu/~jianwu/<http
>://users.sdsc.edu/%7Ejianwu/>
>
>Assistant Project Scientist
>Scientific Workflow Automation Technologies (SWAT) Laboratory
>San Diego Supercomputer Center
>University of California, San Diego
>San Diego, CA, U.S.A.
>
>On 11/1/12 1:45 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE &
>TECHNOLOGY INC] wrote:
>Thanks.  The workflow crashes kepler at a "type detection" step.  So I am
>trying just the SSH session step for starters.  I am running in an
>environment with two-factor authentication.  So I authenticate ssh with
>control master enabled hoping that subsequent ssh attempts by kepler use
>the existing open connection.   At the kepler workflow I set up the ssh
>session with an identity file hoping it would use the control master
>session, but it is failing to do so.  Any thoughts on this mode of
>operation?
>Eduardo
>
>From: Jianwu Wang <jianwu at sdsc.edu<mailto:jianwu at sdsc.edu>>
>Date: Wednesday, October 31, 2012 5:50 PM
>To: "Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE & TECHNOLOGY INC]"
><eduardo.g.valente at nasa.gov<mailto:eduardo.g.valente at nasa.gov>>
>Cc: 
>"kepler-users at kepler-project.org<mailto:kepler-users at kepler-project.org>"
><kepler-users at kepler-project.org<mailto:kepler-users at kepler-project.org>>
>Subject: Re: [kepler-users] Sample workflow using JobSubmission/JobManager
>
>Hi Eduardo,
>
>    A sample workflow using JobSubmission/JobManager can be found at
>https://code.kepler-project.org/code/kepler/trunk/workflows/SC06-Tutorial/
>JobSubmission.xml. To use it in PBS environment,  you just need to edit
>the 'JobManager' parameter to be 'PBS'. Other parameters such as
>'SimTarget' and 'JobScript' also need to be configured to fit your
>information.
>
>Best wishes
>
>Sincerely yours
>
>Jianwu Wang, Ph.D.
>jianwu at sdsc.edu<mailto:jianwu at sdsc.edu>http://users.sdsc.edu/~jianwu/<http
>://users.sdsc.edu/%7Ejianwu/>
>
>Assistant Project Scientist
>Scientific Workflow Automation Technologies (SWAT) Laboratory
>San Diego Supercomputer Center
>University of California, San Diego
>San Diego, CA, U.S.A.
>
>On 10/31/12 2:22 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE &
>TECHNOLOGY INC] wrote:
>Does anyone have a sample workflow that exercises a PBS based HPC
>environment that they would like to share?
>Thanks.
>
>
>
>
>_______________________________________________
>Kepler-users mailing list
>Kepler-users at kepler-project.org<mailto:Kepler-users at kepler-project.org>htt
>p://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>
>
>
>
>