[kepler-users] dataflow using Kepler on Amazon EC2

[Luqman Hodgkinson]

Hello Frank,
Thank you so much for your reply. This sounds very promising. Other software that I have been considering are Taverna (http://www.taverna.org.uk/) and VisTrails (http://www.vistrails.org/index.php/Main_Page). Are you familiar with these other platforms? Do you know of any reason to prefer Kepler for this use case over Taverna or VisTrails? Why in particular did you choose to use Kepler? Would you say that the three platforms are roughly equivalent or are there special features of Kepler that make it better to use than these other platforms? 
		Sincerely, with best wishes,
		LuQman




On Mar 23, 2011, at 3:47 PM, Frank White wrote:

> Hello Lugman:
> I can speak to at least some of what you want to accomplish.  My colleague and I have taken Jianwu Wang's and Jing Tao's fine work on the Kepler Workflow Run engine (see https://kepler-project.org/developers/interest-groups/distributed/technical-documentation/the-overview-of-the-workflow-run-engine-component) and built up an AMI that we regularly run on Amazon EC2.  The AMI is built from base Amazon Fedora core 14 image, onto which we have installed Kepler 2.1, Tomcat6 and Axis2 - all of which are required to run the Kepler Web Service. This enables us to make SOAP/REST calls to execute Kepler .kar files that have been uploaded to a Kepler repository. There were a few difficulties along the way, but with Jing's and Jianwu's help, we were able to make it work quite reliably. On top of that, we have implemented several actors that call other Amazon services, actors that call just plain Java classes, and actors that call Java classes that wrap some fairly complex C++ libraries.
> 
> Overall, we were able to find ample examples of how to do at least some of what you describe. We would be willing to share both the 'recipe' that we documented to build up our Kepler AMI, as well as the AMI itself as long as you understand that we are a small start-up, and have very limited resources to support it.
> 
> Cheers,
> 
> Frank White
> Optensity, Inc.
> 
> 
> Message: 1
> Date: Tue, 22 Mar 2011 15:51:17 -0700
> From: Jianwu Wang<jianwu at sdsc.edu>
> To: Luqman Hodgkinson<luqman at berkeley.edu>
> Cc:kepler-users at kepler-project.org
> Subject: Re: [kepler-users] dataflow using Kepler on Amazon EC2
> Message-ID:<4D8927E5.5030006 at sdsc.edu>
> Content-Type: text/plain; charset=UTF-8; format=flowed
> 
> Hi Luqman,
> 
>     Your target is still not clear to me. Please break it into sub
> tasks so that we can help more efficiently. Or you can try Kepler first
> before getting more specific questions to ask.
> 
>     About Kepler workflow execution on EC2, I did some experiments on
> it and don't think it is hard to execute Kepler workflows on EC2.
> 
> Best wishes
> 
> Sincerely yours
> 
> Jianwu Wang
> jianwu at sdsc.edu
> http://users.sdsc.edu/~jianwu/
> 
> Assistant Project Scientist
> Scientific Workflow Automation Technologies (SWAT) Laboratory
> San Diego Supercomputer Center
> University of California, San Diego
> San Diego, CA, U.S.A.
> 
> 
> On 3/21/2011 5:10 PM, Luqman Hodgkinson wrote:
> 
>> >
>> >
>> >
>> >  Dear Kepler developers,
>> >  I have a collection of Java classes linked by a custom dataflow architecture. All classes are in a single project but some of these classes call executables written in languages other than Java. I am investigating the possibility of transitioning to Kepler. Essentially my desires are to link these Java classes in a DAG representing the dataflow and to execute the dataflow in Amazon EC2. The data flowing along the edges are arbitrary custom Java classes. Additionally it is important to cache intermediate results. The data is acquired from a few web services: iRefIndex, IntAct, UniProt, and Gene Ontology. There are complex software dependencies so after setting up the dataflow I would like to save the entire system as an abstract machine image (AMI). How difficult would this transition be, and would it be worth the effort? I would appreciate your comments and advice.
>> >  		Sincerely, with best wishes,
>> >  		Luqman Hodgkinson,
>> >  		Ph.D. student, UC-Berkeley
>> >  _______________________________________________
>> >  Kepler-users mailing list
>> >  Kepler-users at kepler-project.org
>> >  http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
> ------------------------------
> 
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
> 
> 
> End of Kepler-users Digest, Vol 70, Issue 10
> ********************************************
> 
> -- 
> Frank White
> fwhite at optensity.com
> 781-771-2798
> 
> 
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users