[kepler-dev] Bug 2240: add support for null values to data passing among ports

[Christopher Brooks]

Hi Edward,

One of the Kepler bugs blocking the Kepler release is:
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2240
which is reproduced below.

Basically, Kepler needs to handle missing data because not all
ecological data sets are complete.

The solution to this problem needs to work in PN and SDF.

The Synchronous/Reactive (SR) domain has the notion of absent values.
For example, domains/sr/lib/Absent.java has this class comment:

 This actor outputs absent values.  That is, it produces no tokens,
 and it calls the sendClear() method of the output port on each
 firing.

The sendClear() method is the IOPort.java sendClear() method.
IOPort also has sendClearInside() and broadcastClear().

Do you have any comments?

Below is the text from the bug report:
Below is the text from the bug report:
--start--
Currently ptolemy and kepler do not support passing null values
(sometimes called missing values) among ports, even though this is
common in analytical systems like R and SAS.  The concept of null is
not even defined in the token types. This causes a real problem for
data sources that are sparsely populated, as well as data streams that
result from data integration operations that might produce null
values.  We need to extend the underlying token representation to
include a concept of null values and the actor framework to protect
existing actors that might not know how to handle null values.
Because nulls cannot currently be represented in Kepler, none of the
existing actors support them.  An exception is thrown whenever a
missing value is detected by the EML data source, and workflow
execution ceases.

Bowers and Jones discussed one possible partial solution to this on
IRC, which is summarized here.

1) Override the Token base class to support null values by providing
   two new methods:

   Token.null() sets the token's value to null
   boolean Token.isNull() returns true if the token has been set to
   null

2) Override TypedIOPort to add a new method that takes a boolean
   "dropNull"

    a) by default this could be set to "true" then the existing get()
   method would be reimplemented to call the new one with a default of
   "true" because we can assume that no existing actor written can
   handle null values, since there is no way to pass null values, and
   so existing calls to get() will invisibly drop null values.

    b) if an actor can handle null values, then it passes "false"
    which indicates that the actor knows how to deal with nulls and
    wants to receive them

so the changes to IOPort (or maybe TypedIOPOrt) would be:

IOPOrt.get(channelindex) becomes get(channelIndex, dropNullValues)
and
IOPort.get(channelIndex, int vectorLength) becomes get(channelIndex,
vectorLength, dropNullValues)

so, for example, the new implmenetation of get(channelIndex) would
simply call get(channelIndex, true), so existing actors would not even
notice the change.
--end--

_Christopher