[kepler-dev] Bug 2240: add support for null values to data
Christopher Brooks
cxh at eecs.berkeley.edu
Mon Jan 30 20:56:36 PST 2006
One of the things that will come up next week is the null value issue.
It might be interesting to get the conversation going a little early.
Below are two piece of email from December.
Below is Professor Arne Huseby's comments about missing values:
> I actually implemented support for missing values in my simulation
> program, Riscue, earlier this fall. Normally, when I run an ordinary
> simulation, I would never encounter missing data. However, I also use
> Riscue to process and analyze real statistical data, and such data
> often come with a lot of missing values.
>
> When I load a table of statistical data into Riscue, Riscue generates
> a set of "data nodes", one for each column in the table. I can then
> do a lot of descriptive statistics using these data nodes, e.g.,
> cumulative distribution plots, histograms, scatter plots, regression
> analysis, correlation analysis, etc. I can also do postprocessing of
> the data, i.e, transformations, merging, filtering etc.
>
> In addition to using data nodes as input to such things, data nodes
> can also be used as elements in a model. When I run a simulation on a
> model containing data nodes, Riscue will sample random values from
> the values contained in each of the data nodes. In statistical
> terminology this is usually called "resampling".
>
> Thus, since I now support data nodes with missing values, I also need
> to deal with these missing values in the simulations. Say that I
> e.g., want to simulate a very simple model with only two nodes, A and
> B, where A is a data node, and B is a node taking the sampled value
> from A, and transforming this into some other value, using some
> function f. Thus, when a "good" value, say x, is sampled from A, then
> B gets the value f(x). If on the other hand, a missing value is
> sampled from A, there is a question how this should be treated in B.
> At least in my applications, the natural thing would be to say that B
> gets a missing value too, i.e., that the missing value state is
> propagated through the model.
>
> So, how did I implement support for this?
>
> One obvious way of dealing with this would be to implement support
> for missing values throughout my entire library of mathematical
> functions. With about 400 such classes, this would be a daunting
> task! Moreover, while a few functions may be able to handle missing
> values in some meaningful way, most of these functions would not. So
> it really didn't make sense to go through all the trouble of
> implementing support for something that in 99.99% of the cases would
> not be meaningful after all.
>
> In Riscue each node owns a formula object, which is a tree composed
> of function objects from my function library. So, in order to
> calculate its value, which is done using a method called
> calcObjectValue(), the node asks its formula to do this, at the same
> time passing a list of input edges to the formula. The formula then
> uses the values obtained from the input edges in its calculations and
> responds back to the node with a value. The edges can also have their
> own formulas, and transform values obtained from their respective
> input nodes in a similar fashion.
>
> This design actually provides a very easy way to support missing
> values. Whenever a request for a value is sent from a formula to some
> object (i.e., to an edge or a node), using a method called
> getObjectValue(), two things can happen:
>
> 1) The object has a valid value which is passed back to the formula.
> 2) The object does not have a valid value, and throws a MissingValueException.
>
> This exception is NOT handled by the formula object, so this way I
> avoided having to implement support for this throughout my function
> library. Instead this is handled by the owner of the formula, i.e.,
> either a node or an edge. Handling this exception is trivial since
> this simply means assigning a missing value (encoded in some suitable
> fashion) to the object instead of a valid value.
>
> So the only (well almost...) changes I needed to make in my code, were:
>
> (i) Modifying the getObjectValue()-method so that missing values
> results in a MissingValueException being thrown.
> (ii) Adding a new "catch block" in the calcObjectValue()-method for
> the nodes and edges, handling the MissingValueException.
>
> I don't know if this solution has any relevance to your problem, but
> maybe this will trigger some ideas.
>
> Arne
On 12/13/05, I (cxh) wrote:
>
> Ok, I hacked up the following:
>
> * Token.java now has nil() and isNil() methods
> I went with nil because null is a Java keyword.
>
> The term "missing" is rather appealing since the Token.toString()
> method usually prints out "present". So, it could be modified to
> print out "missing". However, I feel that someone is likely to have
> a parameter named "missing" somewhere. nil seems safer.
>
> This is not cast in stone, comments are welcome.
>
> These methods have tests.
>
> * data/expr/Constants.java now defines a "nil" constant which
> is a Token that has the nil() method:
> ptolemy.data.Token nil = new ptolemy.data.Token();
> nil.nil();
> _table.put("nil", nil);
> Thus, one can now create expressions that have nil in them
>
> The "nil" constant has a test in data/expr/test/PtParser.tcl
>
> * actor/lib/RemoveNilTokens.java:
> A new actor that reads its input and discards any nil tokens
> in the fire() method. It might be better to do this in prefire()
> This actor is available in the "More Libraries" -> "Esoteric" section.
>
> Note that this actor should not be used in SDF.
>
> No tests yet.
>
> * domains/pn/demo/RemoveNilTokens/RemoveNilTokens.xml
> A model that uses RemoveNilTokens.
> I think I'm terminating the PN process poorly. I get 7 outputs
> instead of 5. I could use some help here.
> Also, I have to explicitly set the type of output of the RemoveNilTokens
> actor.
>
>
> The model looks like:
>
> Bool Switch
> Ramp----> |
> |----------------> RemoveNilTokens ---------> Display
> Const | |
> that ---> | --> "Code that stops PN"
> produces _
> nil ^
> |
> |
> Bernoulli -
>
>
> So, now we have a straw man in PN to try out.
>
> Questions:
> 1) Would something like this meet the needs of the Kepler group?
> 2) Should RemoveNilTokens do something in prefire()?
> I'm not sure if I can get access to the Token and call Token.isNil()
> in prefire().
> 3) Is "nil" an ok name?
> 4) How do I get PN to terminate properly after I see 5 non-nil tokens?
>
> _Christopher
>
> Edward wrote:
> --------
>
>
> At 08:16 AM 12/13/2005 -0800, Shawn Bowers wrote:
> >One option for SDF would be to literally "propagate" the missing-valued
> >tokens, instead of "dropping" them (as in PN). Then, it seems the
> >consumption/production rates at least wouldn't be violated. Doing this
> >correctly might be a bit tricky, but could probably perform this based on
> >the token production/consumption rates --
>
> In SDF, before any actor is fired, its prefire() method is called.
> For all our actors that require input data, if prefire() will
> return false if there is no input token, and the actor will not
> be fired. Consequently, its outputs will also have no token...
>
> So SDF already does this propagation...
>
> Edward
>
>
>
> ------------
> Edward A. Lee
> Professor, Chair of the EE Division, Associate Chair of EECS
> 231 Cory Hall, UC Berkeley, Berkeley, CA 94720
> phone: 510-642-0253 or 510-642-0455, fax: 510-642-2845
> eal at eecs.Berkeley.EDU, http://ptolemy.eecs.berkeley.edu/~eal
>
> _______________________________________________
> Ptolemy maillist - Ptolemy at chess.eecs.berkeley.edu
> http://chess.eecs.berkeley.edu/ptolemy/listinfo/ptolemy
> --------
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at ecoinformatics.org
> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
More information about the Kepler-dev
mailing list