[kepler-users] How to run kepler Command Line tool to generate provenance data

[Jianwu Wang]

Hi Madhavi:

     About the webservice question: It should work if you want to use 
the stage 1 operations of Kepler web service.  More information can be 
found at 
https://kepler-project.org/developers/interest-groups/distributed/technical-documentation/the-overview-of-the-workflow-run-engine-component.

Best wishes

Sincerely yours

Jianwu Wang
jianwu at sdsc.edu
http://users.sdsc.edu/~jianwu/

Assistant Project Scientist
Scientific Workflow Automation Technologies (SWAT) Laboratory
San Diego Supercomputer Center
University of California, San Diego
San Diego, CA, U.S.A.


On 1/28/2011 1:02 PM, Derik Barseghian wrote:
> Madhavi,
> Yes this is possible. You may publish your actors (including the jar files) as a suite, and then other users may download and use this suite. Please see:
> https://kepler-project.org/developers/teams/build/documentation/build-system-instructions#making-your-own-modules
> https://kepler-project.org/developers/teams/build/documentation/build-system-instructions#storing-your-own-modules
> https://kepler-project.org/developers/teams/build/documentation/build-system-instructions#Create_non-kepler_svn_repository_suite
>
> I'm not very familiar with the webservice module, Jianwu may have to help with that question.
>
> Derik
>
> On Jan 28, 2011, at 12:47 AM, Madhavi Tikhe wrote:
>
>> Hi Derik,
>>
>> I agree that it is useful to download code while developing new actors. It really helps and easy to look for examples.
>> But once a person is done with writing his own actors, he/she would like to put the actors in production. Or in other words one would like to run those actors in another instance of Kepler which is installed and used by other users. In this case, is it possible to instantiate actors in a different instance of Kepler? What all setup is required to do this? Where to put the  jar files required by my actors?
>>
>> On the same topic,
>>
>> Now if I have created a workflow using my own written actors and now want to run the workflow using Kepler web services. In this case, I would put the jar files required by my actor in the axis2 lib directory and run the workflow. Is this right? Do you have any other thoughts?
>>
>> Thanks,
>> Madhavi
>>
>> -----Original Message-----
>> From: Derik Barseghian [mailto:barseghian at nceas.ucsb.edu]
>> Sent: Friday, January 28, 2011 1:38 AM
>> To: Madhavi Tikhe
>> Cc: kepler-users at kepler-project.org Users
>> Subject: Re: [kepler-users] How to run kepler Command Line tool to generate provenance data
>>
>> Hi Madhavi,
>>
>> I'm not sure about being able to write an actor without downloading the source. Generally it's very useful to be able to refer to the source as you code.
>> Keep in mind some actors may be developed from directly within Kepler by modifying others, e.g. by customizing the Rexpression, MatlabExpression, Python, or External Execution actors, or by creating a Composite.
>>
>> I don't think there's a way to instantiate an actor from command line -- I'm not clear on what you're after here.
>>
>> For information on suite vs module, please see:
>> https://kepler-project.org/developers/teams/build/documentation/build-system-instructions#making-your-own-modules
>> Basically a suite allows you to group suites and/or modules.
>>
>> I don't know of an existing actor that does scatter/gather. You may want to write one using Java NIO: http://tutorials.jenkov.com/java-nio/scatter-gather.html
>>
>> Derik
>>
>> On Jan 26, 2011, at 3:19 AM, Madhavi Tikhe wrote:
>>
>>> Hi Derik,
>>>
>>> Thanks a lot again for the information.
>>> Now that I am getting more n more interested in Kepler, I want to create my own actors and workflow for BFast.
>>> I have gone through the steps creating hello world actor but then it needs to download/compile the Kepler's source code.
>>> Is it possible to write an actor using the kepler's jar files and not using the code?
>>> Is there any command line/non-gui way to instantiate a component?
>>> Also what is the difference between suite and a module?
>>>
>>> Does Kelper provide any way to do scatter-gather?
>>>
>>> Thanks again for your help,
>>> Madhavi
>>>
>>> -----Original Message-----
>>> From: Derik Barseghian [mailto:barseghian at nceas.ucsb.edu]
>>> Sent: Wednesday, January 26, 2011 12:58 AM
>>> To: Madhavi Tikhe
>>> Cc: kepler-users at kepler-project.org
>>> Subject: Re: [kepler-users] How to run kepler Command Line tool to generate provenance data
>>>
>>> Hi Madhavi,
>>>
>>> I'm not aware of a way to have 1 kepler process launch and run 2 or more workflows in a row before quitting. You could simply run a series from command line like:
>>> ant run-w...; ant run-w...; etc.
>>> or write a script to loop through workflow files in a directory, and/or a cron job.
>>>
>>> On a somewhat related note, Jing has added a scheduler module to kepler, which allows you to schedule a workflow to run periodically on a remote kepler server. Any resulting run-kars from these executions are written into a kepler repository (currently MetaCat). This module is not yet released, but will probably be available in the near future.
>>>
>>> Derik
>>>
>>> On Jan 24, 2011, at 9:39 PM, Madhavi Tikhe wrote:
>>>
>>>> Thanks a lot for your reply Derik. It helped.
>>>>
>>>> When we talk about workflow execution from command line, it is possible to execute the two or more workflows in a batch (batch processing of the workflows)?
>>>> Thanks again,
>>>> Madhavi
>>>>
>>>> -----Original Message-----
>>>> From: Derik Barseghian [mailto:barseghian at nceas.ucsb.edu]
>>>> Sent: Tuesday, January 25, 2011 1:02 AM
>>>> To: Madhavi Tikhe
>>>> Cc: kepler-users at kepler-project.org
>>>> Subject: Re: [kepler-users] How to run kepler Command Line tool to generate provenance data
>>>>
>>>> Hi Madhavi,
>>>>
>>>> Yes. The workflow must contain a ProvenanceRecorder, i.e. it must be saved from the GUI while using the provenance suite. Executions of this workflow will then write provenance data.
>>>>
>>>> Additionally, if you would like your execution to generate a "run-kar" containing some metadata about the execution and a report pdf upon completion, from within the reporting suite you should create a report design from the Reporting view and save to a kar, set keepResultantRunKars true in reporting/resources/configurations/configuration.xml, and then execute from the command line.
>>>>
>>>> There are a few ways to execute from command line. One is:
>>>> ant run-workflow-no-gui-server -Dworkflow=/Users/derik/KeplerData/workflows/MyWorkflows/yourWorkflow.kar
>>>>
>>>> The resultant run-kar will be created in KeplerData/workflow-runs
>>>>
>>>> Derik
>>>>
>>>> On Jan 23, 2011, at 10:37 PM, Madhavi Tikhe wrote:
>>>>
>>>>> Hello,
>>>>> I would like to run the Kepler from command line to execute a workflow. In this case is it possible to generate provenance data?
>>>>> Regards,
>>>>> Madhavi
>>>>> DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>>>>>
>>>>> _______________________________________________
>>>>> Kepler-users mailing list
>>>>> Kepler-users at kepler-project.org
>>>>> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>>>
>>>> DISCLAIMER
>>>> ==========
>>>> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>>>
>>> DISCLAIMER
>>> ==========
>>> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.