[kepler-dev] data turbine actor query

[Derik Barseghian]

I see I failed to cc the list on my first reply, so I've pasted it  
below Matt's for anyone interested.

I'll add that I've also developed a DataTurbine Source to parse and  
push into DT data from a Campbell logger, and can help on that end  
too. (If write functionality were added to the DataTurbine actor, I'd  
add this parsing code into another Kepler actor, allowing for  
workflows that encompass 1 more step of our process).

Derik


On May 26, 2009, at 6:37 PM, Matt Jones wrote:

> Hi Stephen,
>
> Derik Barseghian is one of the authors of the DataTurbine actor for
> Kepler.  I'm not sure if he has contacted you yet, but I think he is
> the best contact for its current capabilities.  At this point, we have
> focused on reading data from Data Turbine.  However, we have discussed
> the need to be able to write into a DT server, so any advances in this
> area would be welcome.  There is a Java framework for treating  Data
> Turbine servers as sources and sinks of data, and I think that Chris
> Jones has used this system for writing data into the DT.  I suspect he
> could guide you on how much work is involved in writing a actor that
> could write into Data Turbine.
>
> To address your other question, we are actively working on a scheduled
> workflow execution engine for Kepler that includes the ability to
> process new data as it becomes available, and can display web reports
> when jobs have completed.  We'll announce these capabilities on
> Kepler-dev as they mature.  In addition, two other efforts have
> developed job management capabilities, one by Paul Allen (Science
> Pipes) and one by Tristan King (Hydrant).  More information on these
> efforts and contact details are on the Kepler web site:
> https://kepler-project.org/developers/interest-groups/webui
>
> Matt
>

Hi Stephen,

Yes, the DataTurbine actor is currently just a (in DataTurbine  
terminology) 'Sink', and would be useful to you as a means of easily  
displaying, or manipulating and displaying, data. It would be great if  
it was a 'Source' as well, so that, as you describe, a Kepler user  
could feed it data, we just haven't had the time to add this yet. If  
you're interested in working on adding this functionality to the  
actor, we'll happily accept any contributions, and will help you get  
started developing with Kepler. If you're interested, I can also point  
you to a variety of workflows that use the actor to get a feel for how  
it works.

I'll let Matt respond to you re: Kepler and scheduling.

Derik


> On Thu, May 21, 2009 at 4:14 PM, Stephen Cameron
> <stephen.cameron at utas.edu.au> wrote:
>> Hello,
>>
>> I am intending to make use of Data Turbine for presenting real-time  
>> data
>> streams to visitors to an oceans data portal that we are developing  
>> for the
>> IMOS project (http://www.imos.org.au).
>>
>> I am interested to find out more about the Kepler "Data Turbine  
>> Actor" that
>> is currently under development. I require a means of loading data  
>> into Data
>> Turbine from a variety of different data files (CSV, netCDF) and  
>> would like
>> a configurable generic solution if possible, using the Kepler  
>> framework may
>> be a possible solution. The reported Data Turbine Actor seems to be  
>> a means
>> of getting data from Data Turbine but maybe some work has been done  
>> on the
>> other side as well.
>>
>> I am interested in Kepler from the point of view of setting up  
>> scheduled
>> workflows as well, basically to look for a file in a certain local  
>> or remote
>> location, see if it is a new file or has changed and process any  
>> new data.
>> Is this a sensible use for Kepler?
>>
>> Thanks for your assistance.
>>
>> --
>> Regards
>>
>> Stephen Cameron
>>
>> Data Programmer
>> Integrated Marine Observing System  (IMOS)
>> eMarine Information Infrastructure Project
>> University of Tasmania, Private Bag 21, Hobart, TAS 7001, Australia
>>
>> Tel: +61 3 6226 8507
>> Fax: +61 3 6226 2997
>> Email: stephen.cameron at utas.edu.au
>> URL: http://www.imos.org.au/eMII.html
>>
>> _______________________________________________
>> Kepler-dev mailing list
>> Kepler-dev at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>
>
>
>
> -- 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Matthew B. Jones
> Director of Informatics Research and Development
> National Center for Ecological Analysis and Synthesis (NCEAS)
> UC Santa Barbara
> jones at nceas.ucsb.edu                       Ph: 1-907-523-1960
> http://www.nceas.ucsb.edu/ecoinfo
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev