[kepler-dev] RE: Handle IPCC package in Kepler ---> Solution was found!
tao at nceas.ucsb.edu
Fri Oct 29 18:43:20 PDT 2004
Thanks for the effort! After I changed srb end point to orion machine and
kepler can download the data file(15M) in IPCC package successfully. It
took about one minute.
Because kepler already passes the JVM parameters when we start it, we
don't need change anything in keper.
So kepler can handle IPCC package now!
On Fri, 29 Oct 2004, Bing Zhu wrote:
> Date: Fri, 29 Oct 2004 17:40:53 -0700
> From: Bing Zhu <bzhu at sdsc.edu>
> To: Rod Spears <rods at ku.edu>, berkley at nceas.ucsb.edu, huggins at nceas.ucsb.edu,
> jones at nceas.ucsb.edu, Jing Tao <tao at nceas.ucsb.edu>
> Cc: Seek-Dev <seek-dev at ecoinformatics.org>, Arun <arun at sdsc.edu>
> Subject: RE: Handle IPCC package in Kepler ---> Solution was found!
> Jing and Matt,
> Luckily our SRB teammate, Arun Swaran Jagatheesan, knew how to solve our
> problem with
> big files.
> The problem is caused by the default size of the memory allocation pool,
> which is 2MB by default.
> please check out the following link for -Xms and -Xmx option for java
> So the solution is to increase the size of memory allocation pool for a java
> program. In our case, this needs
> to be done in both client and server side with the following argument.
> $ java -Xms500m -Xmx500m ....
> , which sets the initial size of memory allocation pool to 500 MB.
> I did so for the tomcat server at orion.sdsc.edu. And then I ran client with
> this arguments. I was able to
> get the 16 MB file through our ecogrid 'get' operation as of following.
> $ java -Xms500m -Xmx500m
> org.ecoinformatics.ecogrid.client.EcogridQueryClient get ...
> Please try your client again with the GSH at orion.sdsc.edu:
> -----Original Message-----
> From: Jing Tao [mailto:tao at nceas.ucsb.edu]
> Sent: Friday, October 29, 2004 10:18 AM
> To: jones at nceas.ucsb.edu; huggins at nceas.ucsb.edu; berkley at nceas.ucsb.edu;
> Bing Zhu; Rod Spears
> Cc: Rod Spears
> Subject: Handle IPCC package in Kepler
> Hi, matt:
> The code in kepler for handle IPCC packages is done. In
> EcogridDataCacheItem class, we added some code to handle srb protocol. And
> I Eml200DataSource package, we added a new output type - As Byte
> Array(Maybe change to another name). In this output type, eml200 actor
> will iterate reading a data stream into a fixed byte array and send the
> array to output. This approach can handel a big data set. I tested some
> small size data from srb, it worked fine.
> The biggest issue now is ecogrid can't not handle Deanna's data file in
> srb because the file size issue(about 15 M). We can download small size
> file successfully from srb ecogrid node. We thought this is maybe memorry
> issue, but Bing tried to run sever and client in a super machine (100 G
> memory) still didn't get luck(By the way, Bing said he didn't see any
> outof memory exception). Bing is still keep working on this. I will
> contact him and hope we can figure it out.
> Jing Tao
> National Center for Ecological
> Analysis and Synthesis (NCEAS)
> 735 State St. Suite 204
> Santa Barbara, CA 93101
National Center for Ecological
Analysis and Synthesis (NCEAS)
735 State St. Suite 204
Santa Barbara, CA 93101
More information about the Kepler-dev