[seek-dev] RE: Handle IPCC package in Kepler ---> Solution was found!

[Bing Zhu]

Jing and Matt,

Luckily our SRB teammate, Arun Swaran Jagatheesan, knew how to solve our
problem with
big files.

The problem is caused by the default size of the memory allocation pool,
which is 2MB by default.
please check out the following link for -Xms and -Xmx option for java
command.
http://java.sun.com/j2se/1.3/docs/tooldocs/solaris/java.html


So the solution is to increase the size of memory allocation pool for a java
program. In our case, this needs
to be done in both client and server side with the following argument.
    $ java -Xms500m -Xmx500m ....
, which sets the initial size of memory allocation pool to 500 MB.

I did so for the tomcat server at orion.sdsc.edu. And then I ran client with
this arguments. I was able to
get the 16 MB file through our ecogrid 'get' operation as of following.

    $ java -Xms500m -Xmx500m
org.ecoinformatics.ecogrid.client.EcogridQueryClient get ...

Please try your client again with the GSH at orion.sdsc.edu:
  http://orion.sdsc.edu:8091/ogsa/services/org/ecoinformatics/ecogrid/QueryS
erviceLevelOne

Sincerely,
Bing





-----Original Message-----
From: Jing Tao [mailto:tao at nceas.ucsb.edu]
Sent: Friday, October 29, 2004 10:18 AM
To: jones at nceas.ucsb.edu; huggins at nceas.ucsb.edu; berkley at nceas.ucsb.edu;
Bing Zhu; Rod Spears
Cc: Rod Spears
Subject: Handle IPCC package in Kepler


Hi, matt:

The code in kepler for handle IPCC packages is done. In
EcogridDataCacheItem class, we added some code to handle srb protocol. And
I Eml200DataSource package, we added a new output type - As Byte
Array(Maybe change to another name). In this output type, eml200 actor
will iterate reading a data stream into a fixed byte array and send the
array to output. This approach can handel a big data set. I tested some
small size data from srb, it worked fine.

The biggest issue now is ecogrid can't not handle Deanna's data file in
srb because the file size issue(about 15 M). We can download small size
file successfully from srb ecogrid node. We thought this is maybe memorry
issue, but Bing tried to run sever and client in a super machine (100 G
memory) still didn't get luck(By the way, Bing said he didn't see any
outof memory exception). Bing is still keep working on this. I will
contact him and hope we can figure it out.

Thanks.

Jing

--
Jing Tao
National Center for Ecological
Analysis and Synthesis (NCEAS)
735 State St. Suite 204
Santa Barbara, CA 93101