[seek-dev] Transfer huge data file in EcoGrid

[Peter McCartney]

For what its worth, heres how we've been doing it with web services at
ASU. We created a transfer web service that has two methods. To work,
each participating node in the network has to have a copy of this
service. The main thing we wanted was a way to do third party transfers.


public String pack(SOAPContext myContext, String eml, String userName) -
this is called at the remote source location. It takes the data object
referenced by the eml, packages it into a zip file and puts the zip file
in a "public" location - this location isnt really totally public - its
just accessible by registered nodes within the network. 

public String unpack(SOAPContext myContext, String hostIP, String
zipFileName, String userName) - This is called at the remote
destination. It connects via scp to the hostIP and requests zipFileName
which it then unzips and writes the contents to a local workspace for
userName.

Since we havent graduated to using certificates, userName is just a
string to indicate who's workspace the files belong to. Validation is
done by the client software and all traffic between services is based on
a registered hosts file. That works for small networks like ours, but
probably doesn't scale well for ecogrid. Theres nothing to deal with
separate sessions for a given user other that the usual unique file
identifiers. That would probably be a good idea, but when we originally
started all this web services stuff, we assumed that the client
application would manage sessions keeping the web services much simpler.
 
Peter McCartney (peter.mccartney at asu.edu)
Center for Environmental-Studies
Arizona State University
 


> -----Original Message-----
> From: seek-dev-admin at ecoinformatics.org 
> [mailto:seek-dev-admin at ecoinformatics.org] On Behalf Of Bing Zhu
> Sent: Friday, December 17, 2004 12:13 PM
> To: Jing Tao; kepler-dev at nceas.ucsb.edu; seek-dev at ecoinformatics.org
> Subject: RE: [seek-dev] Transfer huge data file in EcoGrid
> 
> 
> Jing,
> 
> It is not clear to me if we still require to install a 
> grid-ftp server in client side. And also we need to have 
> 'session ID' after we implemented authentication service.
> 
> Bing
> 
> 
> -----Original Message-----
> From: seek-dev-admin at ecoinformatics.org 
> [mailto:seek-dev-admin at ecoinformatics.org]On Behalf Of Jing Tao
> Sent: Thursday, December 16, 2004 2:41 PM
> To: kepler-dev at nceas.ucsb.edu; seek-dev at ecoinformatics.org
> Subject: [seek-dev] Transfer huge data file in EcoGrid
> 
> 
> Hi, everyone:
> 
> It seems we haven't found a way to stream big data object 
> from EcoGrid service to client machine directly.
> 
> Here is a hybrid approach which i am thinking to figure out 
> this issue by use grid ftp.
> 
> Hybrid means this approach will need both client lib and web 
> service. The main task will be taken in client side and web 
> service will provide some info.
> 
> In client lib, we will created a method named "transfer" and 
> it signature will be:
> 
> public void transfer(String host, int port, String docId, 
> DataSink sink)
> 
> This method will initialize a GridFTPClient object(using host 
> and port) and call a "extenedGet" method to get the remote 
> file: extendedGet(String remoteFileName, long size, DataSink sink)
> 
> In order to get remoteFileName and file size, we need two web 
> services in
> EcoGrid:
> String getFilePath(String docid)
> long   getFileSize(String docid)
> 
> The above two web service method will be called in "transfer" 
> method before calling "extenedGet" in order to get 
> remoteFileName and file size.
> 
> This approach is not a pure web service, client software(such 
> as Kepler) should stick the ecogrid client lib. In ecogrid 
> service, we also need to run a grid ftp server.
> 
> Any suggestion and comment will be appreciated.
> 
> Jing
> 
> 
> --
> Jing Tao
> National Center for Ecological
> Analysis and Synthesis (NCEAS)
> 735 State St. Suite 204
> Santa Barbara, CA 93101
> 
> _______________________________________________
> seek-dev mailing list
> seek-dev at ecoinformatics.org 
> http://www.ecoinformatics.org/mailman/listinfo> /seek-dev
> 
> 
> _______________________________________________
> 
> seek-dev mailing list
> seek-dev at ecoinformatics.org 
> http://www.ecoinformatics.org/mailman/listinfo> /seek-dev
>