[kepler-users] dataflow using Kepler on Amazon EC2

[Jianwu Wang]

Hi Frank,

     It's great to know your workflows run well on EC2. If you can 
document and share what you have done, I think it will be definitely 
valuable to other users in Kepler community.

Best wishes

Sincerely yours

Jianwu Wang
jianwu at sdsc.edu
http://users.sdsc.edu/~jianwu/

Assistant Project Scientist
Scientific Workflow Automation Technologies (SWAT) Laboratory
San Diego Supercomputer Center
University of California, San Diego
San Diego, CA, U.S.A.


On 3/23/2011 3:47 PM, Frank White wrote:
> Hello Lugman:
> I can speak to at least some of what you want to accomplish.  My 
> colleague and I have taken Jianwu Wang's and Jing Tao's fine work on 
> the Kepler Workflow Run engine (see 
> https://kepler-project.org/developers/interest-groups/distributed/technical-documentation/the-overview-of-the-workflow-run-engine-component) 
> and built up an AMI that we regularly run on Amazon EC2.  The AMI is 
> built from base Amazon Fedora core 14 image, onto which we have 
> installed Kepler 2.1, Tomcat6 and Axis2 - all of which are required to 
> run the Kepler Web Service. This enables us to make SOAP/REST calls to 
> execute Kepler .kar files that have been uploaded to a Kepler 
> repository. There were a few difficulties along the way, but with 
> Jing's and Jianwu's help, we were able to make it work quite reliably. 
> On top of that, we have implemented several actors that call other 
> Amazon services, actors that call just plain Java classes, and actors 
> that call Java classes that wrap some fairly complex C++ libraries.
>
> Overall, we were able to find ample examples of how to do at least 
> some of what you describe. We would be willing to share both the 
> 'recipe' that we documented to build up our Kepler AMI, as well as the 
> AMI itself as long as you understand that we are a small start-up, and 
> have very limited resources to support it.
>
> Cheers,
>
> Frank White
> Optensity, Inc.
>
>
> Message: 1
> Date: Tue, 22 Mar 2011 15:51:17 -0700
> From: Jianwu Wang<jianwu at sdsc.edu>
> To: Luqman Hodgkinson<luqman at berkeley.edu>
> Cc:kepler-users at kepler-project.org
> Subject: Re: [kepler-users] dataflow using Kepler on Amazon EC2
> Message-ID:<4D8927E5.5030006 at sdsc.edu>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Hi Luqman,
>
>      Your target is still not clear to me. Please break it into sub
> tasks so that we can help more efficiently. Or you can try Kepler first
> before getting more specific questions to ask.
>
>      About Kepler workflow execution on EC2, I did some experiments on
> it and don't think it is hard to execute Kepler workflows on EC2.
>
> Best wishes
>
> Sincerely yours
>
> Jianwu Wang
> jianwu at sdsc.edu
> http://users.sdsc.edu/~jianwu/
>
> Assistant Project Scientist
> Scientific Workflow Automation Technologies (SWAT) Laboratory
> San Diego Supercomputer Center
> University of California, San Diego
> San Diego, CA, U.S.A.
>
>
> On 3/21/2011 5:10 PM, Luqman Hodgkinson wrote:
>
>> >
>> >
>> >
>> >  Dear Kepler developers,
>> >  I have a collection of Java classes linked by a custom dataflow 
>> architecture. All classes are in a single project but some of these 
>> classes call executables written in languages other than Java. I am 
>> investigating the possibility of transitioning to Kepler. Essentially 
>> my desires are to link these Java classes in a DAG representing the 
>> dataflow and to execute the dataflow in Amazon EC2. The data flowing 
>> along the edges are arbitrary custom Java classes. Additionally it is 
>> important to cache intermediate results. The data is acquired from a 
>> few web services: iRefIndex, IntAct, UniProt, and Gene Ontology. 
>> There are complex software dependencies so after setting up the 
>> dataflow I would like to save the entire system as an abstract 
>> machine image (AMI). How difficult would this transition be, and 
>> would it be worth the effort? I would appreciate your comments and 
>> advice.
>> >          Sincerely, with best wishes,
>> >          Luqman Hodgkinson,
>> >          Ph.D. student, UC-Berkeley
>> >  _______________________________________________
>> >  Kepler-users mailing list
>> >  Kepler-users at kepler-project.org
>> >  http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
> ------------------------------
>
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>
>
> End of Kepler-users Digest, Vol 70, Issue 10
> ********************************************
>