Madhavi,<div><br></div><div>Some Kepler users have definitely used Kepler at scales like this -- Norbert's work on plasma fusion simulations that generate 800GB of data in a 30 hour simulation run under Kepler control comes to mind. However, the performance at large scale like this will entirely depend on the specific workflows and actors used and how they handle the data. Any actors that try to pass large data objects like that in memory, or try to process them without considering their scale, will fail. Few of the standard actors have currently been built to deal with that, so you'll need to be careful. As long as you are streaming small chunks of data through the engine at a time, or if you use control tokens to control processing flow on data objects stored outside of memory, or if you use actors that you specially design to work with large data, then you should be good to go. The devil is of course in the details of your particular case.</div>
<div><br></div><div>Matt<br><br><div class="gmail_quote">On Mon, Jan 31, 2011 at 9:02 PM, Madhavi Tikhe <span dir="ltr"><<a href="mailto:madhavi_tikhe@persistent.co.in">madhavi_tikhe@persistent.co.in</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="color:#1F497D">Hi Jianwu,</span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="color:#1F497D">Right, I was referring to Next
Gen Sequencing. One of the problems we face is the data size. It is huge. The file
size can range from 50mb to 500gb. Would you please let me know if you have any
metrics for performance of Kepler when the some or all actors perform
CPU/memory intensive tasks? </span></p>
<p class="MsoNormal"><span style="color:#1F497D">I want to know if Kepler can
gracefully handle such a large data. Pl let me know what your thoughts are on
this.</span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,</span></p>
<p class="MsoNormal"><span style="color:#1F497D">Madhavi</span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;color:windowtext">From:</span></b><span style="font-size:10.0pt;color:windowtext"> Jianwu Wang [mailto:<a href="mailto:jianwu@sdsc.edu" target="_blank">jianwu@sdsc.edu</a>] <br>
<b>Sent:</b> Monday, January 31, 2011 11:50 PM<br>
<b>To:</b> Madhavi Tikhe<br>
<b>Cc:</b> <a href="mailto:kepler-users@kepler-project.org" target="_blank">kepler-users@kepler-project.org</a> Users<br>
<b>Subject:</b> Re: [kepler-users] NGS Specific</span></p>
</div>
</div><div><div></div><div class="h5">
<p class="MsoNormal"> </p>
<p class="MsoNormal">Hi Madhavi,<br>
<br>
I guess NGS means 'Next Generation Sequencing'. I know one
work related to it in Kepler is to use Hadoop to support. You can find our
paper, titled: "Kepler + Hadoop : A General Architecture Facilitating
Data-Intensive Applications in Scientific Workflow Systems" at my website <a href="http://users.sdsc.edu/~jianwu/" target="_blank">http://users.sdsc.edu/~jianwu/</a>. Notes
that this module is available in Kepler trunk, but not in production yet. We
are updating this module recently.<br>
<br>
</p>
<pre>Best wishes</pre><pre> </pre><pre>Sincerely yours</pre><pre> </pre><pre>Jianwu Wang</pre><pre><a href="mailto:jianwu@sdsc.edu" target="_blank">jianwu@sdsc.edu</a></pre><pre><a href="http://users.sdsc.edu/~jianwu/" target="_blank">http://users.sdsc.edu/~jianwu/</a></pre>
<pre> </pre><pre>Assistant Project Scientist</pre><pre>Scientific Workflow Automation Technologies (SWAT) Laboratory</pre><pre>San Diego Supercomputer Center </pre><pre>University of California, San Diego</pre><pre>San Diego, CA, U.S.A. </pre>
<p class="MsoNormal"><br>
On 1/31/2011 5:45 AM, Madhavi Tikhe wrote: </p>
<p class="MsoNormal">Hi</p>
<p class="MsoNormal">What all NGS specific features does Kepler provide?</p>
<p>Support for large data
sets ( 1tb)?</p>
<p>Support for CPU intensive
modules?</p>
<p>Any other features?</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks,</p>
<p class="MsoNormal">Madhavi</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p>DISCLAIMER ========== This e-mail may contain privileged and confidential
information which is the property of Persistent Systems Ltd. It is intended
only for the use of the individual or entity to which it is addressed. If you
are not the intended recipient, you are not authorized to read, retain, copy,
print, distribute or use this message. If you have received this communication
in error, please notify the sender and delete all copies of this message. Persistent
Systems Ltd. does not accept any liability for virus infected mails.</p>
<pre> </pre><pre> </pre><pre>_______________________________________________</pre><pre>Kepler-users mailing list</pre><pre><a href="mailto:Kepler-users@kepler-project.org" target="_blank">Kepler-users@kepler-project.org</a></pre>
<pre><a href="http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users" target="_blank">http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users</a></pre></div></div></div><div><div></div><div class="h5">
<p>DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.</p>
</div></div></div>
<br>_______________________________________________<br>
Kepler-users mailing list<br>
<a href="mailto:Kepler-users@kepler-project.org">Kepler-users@kepler-project.org</a><br>
<a href="http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users" target="_blank">http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users</a><br>
<br></blockquote></div><br></div>