[kepler-users] Using Kepler on a Rocks cluster

[Sean Riddle]

I just recently finished working on a project called WATERS (Workflow  
for the Alignment, Taxonomy, and Ecology of Ribosomal Sequences), and  
it also runs jobs on a Rocks cluster. It uses the Comad framework (so  
data is managed as a stream that passes sequentially through all the  
actors, each actor choosing what to add or remove), so it might be a  
bit unfamiliar, but it could be worth a look.

I handled the Rocks cluster submission and monitoring through qsub  
etc., but I do it in the Java code rather than at the actor level, so  
that it can be abstracted away and hidden from the user.

A zip intended for end users can be found at daks.ucdavis.edu/ 
waters.zip. I believe the source is included. Hope it helps.

- Sean

Sent from my iPhone

On Dec 2, 2009, at 9:26 AM, Jianwu Wang <jianwu at sdsc.edu> wrote:


> Arthur Goldberg wrote:
>>
>> Hello
>>
>> I've a bio-informatics workflow, that I'm considering running in  
>> Kepler. The workflow analyzes about a dozen genomes, and will take  
>> a few hundred hours of computing, which I'll run on our 64 node  
>> Rocks cluster.
>>
>> Several Perl programs retrieve the data and analyze it, inputting  
>> and outputting results from and to both files and a MySQL database.  
>> Each program needs to be instantiated multiple times with different  
>> input.
>>
>> It appears that Kepler might be able to execute these computations  
>> with its PN scheduler and ExternalExecution Actor, but 1) we  
>> typically manage jobs on the cluster via the SGE (qsub) and 2)  
>> Kepler's documentation says that "to use the ExternalExecution  
>> actor, the invoked application must be on the local computer",  
>> which implies that Kepler must be installed on all nodes of the  
>> cluster.
>>
>> How would one approach this problem with Kepler?
>>
>> If Kepler isn't the right tool for this problem, what would you  
>> recommend?
>>
>> Thanks
>> Arthur
>>
>>
>> Arthur P. Goldberg, PhD
>>
>> Research Scientist in Bioinformatics Group
>> Plant Systems Biology Laboratory
>> www.virtualplant.org
>>
>> Visiting Academic
>> Computer Science Department
>> Courant Institute of Mathematical Sciences
>> www.cs.nyu.edu/artg
>>
>> artg at cs.nyu.edu
>> New York University
>> 212 995-4918
>> 100 Washington Sq East
>> 8th Floor Silver Building
>>
>> _______________________________________________
>> Kepler-users mailing list
>> Kepler-users at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>
>
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nceas.ucsb.edu/kepler/pipermail/kepler-users/attachments/20091202/2d93a244/attachment.html>