[kepler-dev] [Bug 5685] data isn't always chunked properly

[bugzilla-daemon at ecoinformatics.org]

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5685

--- Comment #3 from Derik Barseghian <barseghian at nceas.ucsb.edu> 2012-08-25 02:17:06 PDT ---
Doesn't look like trolling through the archive will be necessary. The same
problem of a datapackage containing two data with two different sampling rates
occurred tonight. I did not receive any errors when adjusting sensor sampling
rates in Kepler. Here are the recents of the last archive workflow run:
-----------------------------------
Sensor Name: gpp-data/CR800_Batt_Volt
Document URL:
http://dev2.nceas.ucsb.edu/knb/metacat?action=read&qformat=default&docid=doc.13458810777906caa4227-8260-449b-8fff-11c07741dcc4.1
Time Range: 2012-08-25 02:21:35 ~ 2012-08-25 02:48:58
Number of Records: 1641

Sensor Name: gpp-data/CR800_Batt_Volt
Document URL:
http://dev2.nceas.ucsb.edu/knb/metacat?action=read&qformat=default&docid=doc.1345881206554a70e25dc-e965-4430-9b17-f99499aff969.1
Time Range: 2012-08-25 02:49:30 ~ 2012-08-25 07:50:36
Number of Records: 602

Sensor Name: gpp-data/CR800_sq311_1
Document URL:
http://dev2.nceas.ucsb.edu/knb/metacat?action=read&qformat=default&docid=doc.1345881330868470d3007-65e6-46c3-997d-5d27e37f15ee.1
Time Range: 2012-08-25 02:21:34 ~ 2012-08-25 02:50:50
Number of Records: 826

Sensor Name: gpp-data/CR800_sq311_1
Document URL:
http://dev2.nceas.ucsb.edu/knb/metacat?action=read&qformat=default&docid=doc.13458814576456ba0d71b-1962-4034-9ee0-6bc8fb41c29b.1
Time Range: 2012-08-25 03:01:05 ~ 2012-08-25 07:41:06
Number of Records: 29

Sensor Name: gpp-data/CR800_sq311_2
Document URL:
http://dev2.nceas.ucsb.edu/knb/metacat?action=read&qformat=default&docid=doc.13458815819617734014f-b2ab-4bd2-85d4-b7a059ca2aa5.1
Time Range: 2012-08-25 02:21:33 ~ 2012-08-25 07:50:36
Number of Records: 356
-----------------------------------

If you look at the raw data for sq311_2, you'll see data at 30s, and then an
un-smooth change to 60s:
-----------------------------------
2012-08-25 02:47:33    0.67334365845
2012-08-25 02:48:03    0.67333245277
2012-08-25 02:48:33    0.67334610224
2012-08-25 02:50:30    0.67336404324
2012-08-25 02:51:35    0.67335760593
2012-08-25 02:52:35    0.67335271835
-----------------------------------

Below is the result of dumping the metadata channels from the DT. You can see
CR800_sq311_2 does only have one metadata entry. 
-----------------------------------
someData.length:3
times.length:3
i:0 someData[0]:CR800_Batt_Volt
altitude=0.000000,coefficients=,conversion-type=no
conversion,daq-method=,isOn=true,latitude=34.412291,longitude=-119.842335,measurement-unit=Volts,sampleMethod=average,samples-per-measurement=1,samplingPeriod=1,sensor-make=Campbell
Scientific,sensor-measurement=,sensor-model=,serial-number=
i:0 times[0]:1.345870971714E9
i:0 someData[1]:CR800_Batt_Volt
altitude=0.000000,coefficients=,conversion-type=no
conversion,daq-method=,isOn=true,latitude=34.412291,longitude=-119.842335,measurement-unit=Volts,sampleMethod=average,samples-per-measurement=1,samplingPeriod=30,sensor-make=Campbell
Scientific,sensor-measurement=,sensor-model=,serial-number=
i:0 times[1]:1.345888139714E9
i:0 someData[2]:CR800_Batt_Volt
altitude=0.000000,coefficients=,conversion-type=no
conversion,daq-method=,isOn=true,latitude=34.412291,longitude=-119.842335,measurement-unit=Volts,sampleMethod=average,samples-per-measurement=1,samplingPeriod=30,sensor-make=Campbell
Scientific,sensor-measurement=,sensor-model=,serial-number=
i:0 times[2]:1.345888200714E9
someData.length:1
times.length:1
i:1 someData[0]:CR800_sq311_1
altitude=0.000000,coefficients=,conversion-type=no
conversion,daq-method=,isOn=true,latitude=34.412291,longitude=-119.842335,measurement-unit=mV,sampleMethod=average,samples-per-measurement=1,samplingPeriod=2,sensor-make=Apogee
Instruments,sensor-measurement=Photosynthetic Photon Flux
(PPF),sensor-model=SQ-311 (sun),serial-number=1612
i:1 times[0]:1.345870973714E9
someData.length:1
times.length:1
i:2 someData[0]:CR800_sq311_2
altitude=0.000000,coefficients=,conversion-type=no
conversion,daq-method=,isOn=true,latitude=34.412291,longitude=-119.842335,measurement-unit=mV,sampleMethod=average,samples-per-measurement=1,samplingPeriod=30,sensor-make=Apogee
Instruments,sensor-measurement=Photosynthetic Photon Flux
(PPF),sensor-model=SQ-311 (sun),serial-number=1609
i:2 times[0]:1.345870973714E9
-----------------------------------

So at least we can rule out the archival workflow.

We should look at how the Sensor actor is inserting metadata entries to DT, if
it can verify they get inserted, etc.

-- 
Configure bugmail: http://bugzilla.ecoinformatics.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA Contact for the bug.