[seek-dev] RE: pgm_cut program
Dave Vieglais
vieglais at KU.EDU
Tue Jul 13 11:56:05 PDT 2004
David Stockwell wrote:
> Hi,
> Yes, I am happy to work with Chad on this, and a third party service
> would work fine too, as downloading a GIS is not a frequent operation.
> pgm_cut is a 10 line prgm that solves our immediate problem of getting
> data out of gigabyte coverages. Similarly the coverages are designed to
> be the easiest possible format to work with without the different
> formats, extents, projections, etc.
pgm is an integer (1 or 2 byte) format that severely limits resolution
of data values at a particular grid cell. It has no inherent spatial
metadata, and so a seperate system must be constructed to handle this.
GeoTIFF is a far more widely used spatial data format that can represent
cells as integer or floating point values, and includes useful
information about the projection and region represented by the data.
Read/write access performance to uncompressed TIFF is similar or
superior to that for pgm.
>
> WhyWhere does the same thing as GARP only different algorithm, so
> anything done with it can be readily applied.
Sure WhyWhere is another method for attempting to describe an ecological
niche, but is not really comparable to GARP or other systems such as
GRASP which seem to be able to discern niche space with a fairly decent
resolution? Why Where seems to be most helpful because it is fast.
Have there been any published comparisons between the niche resolving
power of Why Where compared to some of the other more commonly used
mechanisms?
>As a goal,
> could I suggest that we look at the capacity for extracting data
> from a number of sources in an integrated way:
> 1. MODIS data repository.
This requires some reasonably complex spatial reprojection and other
operations to retrieve continuous coverages. It is not really something
that should be done on the fly (not scalable).
> 2. global ecosystems NOAA/NGDC data
> 3. LifeMapper
Not sure what you mean here. Are you looking for point values or
distribution estimates?
>
> We would need:
> 1. Metadata search that filters possible coverages related to an
> ecological concept.
> 2. Third party service.
> Cheers
>
The tasks you mention basically describe a portion of the ecogrid.
Instead of building a solution for one particular application (why
where), it seems to make more sense to put a little more effort into
producing a significantly more generic solution that utilizes existing
standards for spatial data sets.
>
> Matt Jones wrote:
>
>> Hi,
>>
>> Also, Chad has implemented several important GIS functions by wrapping
>> GRASS libraries behind web services rather than using GDAL directly.
>> These services can be called by Kepler and other clients. The idea is
>> that a workflow could be constructed that would first query to locate
>> data, then do the various clipping and reprojections needed directly
>> at the source, then transfer the clipped data to whatever analytical
>> service needs it in the workflow. This use of what we've been calling
>> 'third party data transfer' and sometimes 'passing data by reference'
>> is going to be an important component of data handling. Ultimately, by
>> exposing the actual data query and manipulation plan to the workflow
>> engine, we will be able to design a scheduler and optimizer that can
>> figure out what the most efficient way to execute the workflow is
>> (where to do what). The workflow engine becomes a controller for a
>> variety of distributed data access, data manipulation, and analysis
>> tasks.
>>
>> So, I think we should be careful to plan out an end-to-end strategy
>> for all data manipulation, rather than doing it piece by piece.
>> Dave's suggestion to use GDAL I think is the right approach in that it
>> is more generic and more broadly useful than just adding on a specific
>> SRB extension. Dave -- could Chad's web-service wrappers around GRASS
>> serve in the same function?
>>
>> Matt
>>
>> David Stockwell wrote:
>>
>>> Hi,
>>> pgm_cut.c is a client program that just implements the srb library
>>> fread and fseek routines. I thought about implementing more
>>> functions on the server side but that is a much bigger project.
>>>
>>> There is an argument of just having a cut function on the
>>> server side, as a cut (or crop) must reduce the data to be
>>> transferred. Some other operations like rescaling may
>>> increse the size of the data to be transferred, and besides
>>> may put a lot of load on the server.
>>>
>>> GDAL seems useful, but it may make sense to modify
>>> it so it could be used on the client side to make simple
>>> SRB (or ecogrid) calls. I have been hoping for something
>>> like this on the server side but I am not in a position to
>>> modify the SRB.
>>>
>>> Cheers
>>>
>>>
>>> Dave Vieglais wrote:
>>>
>>>> Hi Bing and Dave,
>>>> I was wondering if you had considered using tools more appropriate
>>>> for handling geospatial data? The pgm libraries handle image data
>>>> ok, but spatial raster data sets have a number of nuances that
>>>> require additional processing on top of merely extracting pixel
>>>> values. One library in particular that is applicable to a wide
>>>> variety of raster data formats is GDAL (geospatial data abstraction
>>>> library). Integrating this with the SRB and as an EcoGrid service
>>>> would provide access to a huge variety and volume of spatial data
>>>> available from various sources, as well as the pgm images. That
>>>> library also has numerous methods and examples for windowing,
>>>> scaling and reprojecting raster data, and so should be fairly easy
>>>> to integrate in the same manner as you have described below. It
>>>> would also provide the significant advantage of providing the
>>>> foundation of a more standardized mechanism for accessing raster
>>>> spatial data through the EcoGrid- something that will be useful to a
>>>> much larger audience compared with the very limited data accessible
>>>> for the "Why Where" system.
>>>>
>>>> cheers,
>>>> Dave V.
>>>>
>>>> Bing Zhu wrote:
>>>>
>>>>> David,
>>>>>
>>>>> I am glad that the SRB implementation works fine in getting partial
>>>>> images
>>>>> for Niche
>>>>> Modeling pgm data.
>>>>>
>>>>> This is a good step. I just had a meeting with Raja discussing
>>>>> about moving
>>>>> this approach to our Ecogrid. Raja suggested to modify the SRB code in
>>>>> Ecogrid
>>>>> node to handle the parameters currently implemented in pgm_cut.
>>>>>
>>>>> With this approach, the URL used by Niche Modeling in calling our
>>>>> Ecogrid's
>>>>> 'get' function
>>>>> to retrieve partial pgm image will have the syntax similar to a cgi
>>>>> request.
>>>>> e.g.
>>>>> $ java org.ecoinformatics.ecogrid.client.EcogridQueryClient get \
>>>>>
>>>>> srb://../home/whywhere/.../n00an1.10.pgm?left=10&top=10&width=25&height=30
>>>>> \
>>>>>
>>>>> http://orion.sdsc.edu:8080/ogsa/services/org/ecoinformatics/ecogrid/EcoGridL
>>>>>
>>>>> evelI
>>>>>
>>>>> One of my next tasks in SEEK will be modify the SEEK SRB code to
>>>>> handle this
>>>>> for Niche Modeling data. And this approach is also applicable for
>>>>> other
>>>>> applications which
>>>>> request partial data transfer.
>>>>>
>>>>> Bing
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: hwliu [mailto:hwliu at sdsc.edu]
>>>>> Sent: Monday, July 12, 2004 12:59 PM
>>>>> To: bzhu at sdsc.edu
>>>>> Cc: davids at sdsc.edu
>>>>> Subject: RE: pgm_cut program
>>>>>
>>>>>
>>>>> Hi, Bing,
>>>>> Thanks for your help. I've just tried the program and it worked
>>>>> well.
>>>>> But, I don't understand the need to use S-command. I just copied
>>>>> pgm_cut to
>>>>> my local directory on landscape and it worked even if I don't copy the
>>>>> S-command directory. By the way, if I want to rebuild the program on
>>>>> Windows, do I need to download the whole directory of SRB1.2.1?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Haowei
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Bing Zhu [mailto:bzhu at sdsc.edu]
>>>>> Sent: Monday, July 12, 2004 10:56 AM
>>>>> To: hwliu
>>>>> Cc: davids at sdsc.edu
>>>>> Subject: RE: pgm_cut program
>>>>>
>>>>> Hi Haowei,
>>>>>
>>>>> I changed the permission for some directories and files.
>>>>> Please try it again and let me know if the permission problem still
>>>>> exists.
>>>>>
>>>>> The files are in the directory, /export/home/bzhu/pgm_cut.
>>>>>
>>>>> You might also need to use SRB software which is installed in
>>>>> /export/home/bzhu/SRB2.1.2. Basically, you just need to use
>>>>> SRB client programs, S-commands, which can be found
>>>>> in /export/home/bzhu/SRB2.1.2/utilities/bin. To use them, you need
>>>>> an SRB account. Currently we can use the user, 'whywhere'.
>>>>> I will be glad to show you how to use SRB software so that
>>>>> you can test the software with different pgm files stored
>>>>> under 'whywhere'.
>>>>>
>>>>> Sincerely,
>>>>> Bing
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: hwliu [mailto:hwliu at sdsc.edu]
>>>>> Sent: Monday, July 12, 2004 10:05 AM
>>>>> To: bzhu at sdsc.edu
>>>>> Cc: davids at sdsc.edu
>>>>> Subject: RE: pgm_cut program
>>>>>
>>>>>
>>>>> Hi, Bing:
>>>>> I will test the new version of pgm_cut, but I don't have the
>>>>> permission
>>>>> to access the directory. Is it possible for you to mail me pgm_cut?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Haowei
>>>>>
>>>>> -----Original Message-----
>>>>> From: Bing Zhu [mailto:bzhu at sdsc.edu]
>>>>> Sent: Friday, July 09, 2004 5:48 PM
>>>>> To: davids at sdsc.edu
>>>>> Cc: Seek-Dev; Arcot Rajasekar; Matt Jones
>>>>> Subject: pgm_cut program
>>>>>
>>>>> David,
>>>>>
>>>>> I modified the pgm_cut.c code to read pgm files stored in SRB
>>>>> storage place.
>>>>>
>>>>> The modified pgm_cut.c can be found in the directory,
>>>>> /export/home/bzhu/pgm_cut,
>>>>> in landscape machine.
>>>>>
>>>>> Here is an example of running 'pgm_cut' to get partial image from a
>>>>> pgm file
>>>>> in SRB.
>>>>>
>>>>> $ pgm_cut 0 0 20 20
>>>>> /home/whywhere.seek/ei/Data/Terrestrial/n00an1.10.pgm
>>>>>
>>>>> Note that I have re-uploaded whywhere data using a new SRB account,
>>>>> 'whywhere',
>>>>> as we discussed last time. The new uploaded data, along with their
>>>>> metadata,
>>>>> is in the collection, /home/whywhere.seek/ei/Data/Terrestrial.
>>>>>
>>>>> Here is the whywhere account info.
>>>>> user name: whywhere
>>>>> password: I will send you in another mail.
>>>>> SRB server: srb.sdsc.edu
>>>>> SRB domain: seek
>>>>> SRB port: 6613
>>>>>
>>>>> I'd like to test it next Monday. Will you be in office next Monday?
>>>>>
>>>>> Sincerely,
>>>>> Bing
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> seek-dev mailing list
>>>>> seek-dev at ecoinformatics.org
>>>>> http://www.ecoinformatics.org/mailman/listinfo/seek-dev
>>>>>
>>>
>>
>
>
More information about the Seek-dev
mailing list