[kepler-users] Sample workflow using JobSubmission/JobManager
Jianwu Wang
jianwu at sdsc.edu
Mon Nov 5 17:11:15 PST 2012
Hi Eduardo,
I agree with you. To make it working, Kepler either need to share
in the externally established SSH connection or establish the master
connection by itself. Because of the limitation of the underlying JSCH
library, I don't think SSH Session actor support the first option. The
second option has to answer two passwords prompted by different screen
questions. The current SSH Session actor only popup a dialogue for user
password. I think Kepler code can be extended to support two password
interaction. Related classes are org.kepler.ssh.SshSession (especially
MyUserInfo inner class).
But I don't have access to resources that need two password
interaction. So it's hard for me to update the code. Are you interested
in getting your hands dirty on updating Kepler code? :)
Best wishes
Sincerely yours
Jianwu Wang, Ph.D.
jianwu at sdsc.edu
http://users.sdsc.edu/~jianwu/
Assistant Project Scientist
Scientific Workflow Automation Technologies (SWAT) Laboratory
San Diego Supercomputer Center
University of California, San Diego
San Diego, CA, U.S.A.
On 11/5/12 2:18 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE &
TECHNOLOGY INC] wrote:
> Hi Jianwu,
>
> The ssh connection is first set up by answering an RSA token
> password followed by a system password. The RSA token based password
> changes every few seconds. Since there are many steps (commands) to
> the remote workflow execution we can't have every connection prompt
> for these dynamic passwords. Therefore it must use the
> preexisting/authenticated connection. With ssh the first connection
> becomes the master and subsequent ones go straight to the prompt. You
> can try that yourself. But with Kepler it would have to either
> establish the master, or use the existing system one. If it were to
> use the system connection there is no password to provide. If it were
> to establish the master connection it would have to answer two
> passwords prompted by different screen questions.
> The bottom line is that Kepler does not seem to share in the
> externally established SSH connection. I will have to research the
> JSCH library for more information on this mode of operation as well.
> Thanks,
> Eduardo
>
> From: Jianwu Wang <jianwu at sdsc.edu <mailto:jianwu at sdsc.edu>>
> Date: Thursday, November 1, 2012 8:12 PM
> To: "Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE & TECHNOLOGY
> INC]" <eduardo.g.valente at nasa.gov <mailto:eduardo.g.valente at nasa.gov>>
> Cc: Norbert Podhorszki <pnorbert at ornl.gov <mailto:pnorbert at ornl.gov>>,
> "kepler-users at kepler-project.org
> <mailto:kepler-users at kepler-project.org>"
> <kepler-users at kepler-project.org <mailto:kepler-users at kepler-project.org>>
> Subject: Re: [kepler-users] Sample workflow using JobSubmission/JobManager
>
> Hi Eduardo,
>
> If you start kepler from kepler.sh/kepler.bat, not double-clicking
> kepler icon, a console will show up and it normally has more
> information on errors.
>
> I checked the '-M' option of ssh and the implementation of ssh
> related code in Kepler. I don't think Kepler supports control master.
> I also don't see the Jsch library we used for ssh actors support it.
> So it's hard for us to update our code to do it.
>
> But I think there is still one way workaround. How do you set up
> the ssh connection at the first place? Type your two passwords in
> command line? Does it work if you try it in Kepler using 'SSH Session'
> actor? The same ssh session actor generated by the 'SSH Session' actor
> can be postponed and shared by other workflow executions within the
> same JVM. To do it, the parameter 'postpone' of the actor has to be
> selected (true) and the 'closeAtEnd' has to be false. So if you split
> the workflow into two workflows. The first workflow only has 'Host'
> and 'SSH Session'. The second one has other parts. If you can generate
> correct ssh session using the first workflow. You can run the second
> workflow many times without creating new ssh sessions. My tests work here.
>
> I'm ccing the email to Norbert. He is the original developer of
> the actors and workflows. He might have better solutions for you.
> Best wishes
>
> Sincerely yours
>
> Jianwu Wang, Ph.D.
> jianwu at sdsc.eduhttp://users.sdsc.edu/~jianwu/
>
> Assistant Project Scientist
> Scientific Workflow Automation Technologies (SWAT) Laboratory
> San Diego Supercomputer Center
> University of California, San Diego
> San Diego, CA, U.S.A.
> On 11/1/12 2:08 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE &
> TECHNOLOGY INC] wrote:
>> The failure is graceful in that the session return true for the
>> failed port. If there is a way to view logs for that actor please
>> let me know.
>>
>> Two factor authentication is the use of two passwords in the
>> admission process one of which utilizes an RSA token (changes every
>> 30 seconds or so).
>>
>> This means that it becomes impractical to automate processes if every
>> time we ssh the password is different. And this two factor
>> authentication cannot be bypassed with public keys (otherwise known
>> as passwordless ssh). The only option left is to use an existing ssh
>> connection enabled as the control master. Generally the –M switch of
>> ssh clients. But it would appear the java ssh client and the system
>> ssh client do not "see" each other. If that is the case than the
>> java version would need such a mode as well. Currently I see two
>> modes: interactive with password request and passwordless with
>> identity file.
>>
>> I cannot otherwise provide you a means to recreate the environment we
>> have. But look into the concept of control master and you will be
>> able to investigate this possibility with the ssh session actor.
>>
>> Thanks,
>> Eduardo
>>
>> From: Jianwu Wang <jianwu at sdsc.edu <mailto:jianwu at sdsc.edu>>
>> Date: Thursday, November 1, 2012 4:52 PM
>> To: "Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE & TECHNOLOGY
>> INC]" <eduardo.g.valente at nasa.gov <mailto:eduardo.g.valente at nasa.gov>>
>> Cc: "kepler-users at kepler-project.org
>> <mailto:kepler-users at kepler-project.org>"
>> <kepler-users at kepler-project.org
>> <mailto:kepler-users at kepler-project.org>>
>> Subject: Re: [kepler-users] Sample workflow using
>> JobSubmission/JobManager
>>
>> Hi Eduardo,
>>
>> So you failed even just using 'SSH Session' actor? Did you get
>> any error or exception message? It's new to me for the two-factor
>> authentication. If you tell me how to reproduce it, I can dig into it
>> and check what went wrong.
>> Best wishes
>>
>> Sincerely yours
>>
>> Jianwu Wang, Ph.D.
>> jianwu at sdsc.eduhttp://users.sdsc.edu/~jianwu/
>>
>> Assistant Project Scientist
>> Scientific Workflow Automation Technologies (SWAT) Laboratory
>> San Diego Supercomputer Center
>> University of California, San Diego
>> San Diego, CA, U.S.A.
>> On 11/1/12 1:45 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE &
>> TECHNOLOGY INC] wrote:
>>> Thanks. The workflow crashes kepler at a "type detection" step. So
>>> I am trying just the SSH session step for starters. I am running in
>>> an environment with two-factor authentication. So I authenticate
>>> ssh with control master enabled hoping that subsequent ssh attempts
>>> by kepler use the existing open connection. At the kepler workflow
>>> I set up the ssh session with an identity file hoping it would use
>>> the control master session, but it is failing to do so. Any
>>> thoughts on this mode of operation?
>>> Eduardo
>>>
>>> From: Jianwu Wang <jianwu at sdsc.edu <mailto:jianwu at sdsc.edu>>
>>> Date: Wednesday, October 31, 2012 5:50 PM
>>> To: "Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE & TECHNOLOGY
>>> INC]" <eduardo.g.valente at nasa.gov <mailto:eduardo.g.valente at nasa.gov>>
>>> Cc: "kepler-users at kepler-project.org
>>> <mailto:kepler-users at kepler-project.org>"
>>> <kepler-users at kepler-project.org
>>> <mailto:kepler-users at kepler-project.org>>
>>> Subject: Re: [kepler-users] Sample workflow using
>>> JobSubmission/JobManager
>>>
>>> Hi Eduardo,
>>>
>>> A sample workflow using JobSubmission/JobManager can be found at
>>> https://code.kepler-project.org/code/kepler/trunk/workflows/SC06-Tutorial/JobSubmission.xml.
>>> To use it in PBS environment, you just need to edit the
>>> 'JobManager' parameter to be 'PBS'. Other parameters such as
>>> 'SimTarget' and 'JobScript' also need to be configured to fit your
>>> information.
>>> Best wishes
>>>
>>> Sincerely yours
>>>
>>> Jianwu Wang, Ph.D.
>>> jianwu at sdsc.eduhttp://users.sdsc.edu/~jianwu/
>>>
>>> Assistant Project Scientist
>>> Scientific Workflow Automation Technologies (SWAT) Laboratory
>>> San Diego Supercomputer Center
>>> University of California, San Diego
>>> San Diego, CA, U.S.A.
>>> On 10/31/12 2:22 PM, Valente, Eduardo G. (GSFC-610.3)[GLOBAL SCIENCE
>>> & TECHNOLOGY INC] wrote:
>>>> Does anyone have a sample workflow that exercises a PBS based HPC
>>>> environment that they would like to share?
>>>> Thanks.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Kepler-users mailing list
>>>> Kepler-users at kepler-project.orghttp://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nceas.ucsb.edu/kepler/pipermail/kepler-users/attachments/20121105/ee804338/attachment-0001.html>
More information about the Kepler-users
mailing list