[kepler-dev] nil problem with MatrixTokens

[Christopher Brooks]

I'm updating the docs for the Ptolemy release and came across a bug
where the cast expression language function would result in bogus data
if the argument was a nil token.

The cast expression language function takes two arguments and converts
the second token to the type of the first.

The cast expression language function is defined in 
data.expr.UtilityFunctions.java:
    /** Convert the second token to the type of the first.
     *  @exception IllegalActionException If the token cannot be converted.
     */
    public static Token cast(Token token1, Token token2)
            throws IllegalActionException {
        return token1.getType().convert(token2);
    }

To use the cast function, start up the Ptolemy Expression Evaluator
by doing file -> New -> ExpressionEvaluator

These are all ok:
 >> cast(int,nil)
 nil
 >> cast({int},nil)
 {nil}
 >> cast(double,nil)
 nil
 >> cast(long,nil)
 nil

These are bogus.  
 >> cast([int],nil)
 [2147483647]
 >> cast([double],nil)
 [NaN]
 >> cast([long],nil)
 [9223372036854775807L]

The problem is that we are getting the values of the nil token for
int, double and long.  MatrixTokes are wrappers for matrices
of the native type, which cannot represent a nil token.  For
example, DoubleMatrixToken has a matrix of Java doubles at its core
and Java doubles cannot represent nil Double tokens.

I think an aside about nil Doubles is warranted:

Note that while it is tempting to have a nil Double have the value
NaN, I don't think this is right.

NaN is not a number, which is not the same as not having a value.  It
is sort of like the difference between 0 and the empty set.

Say I was taking the averages of 30 temperatures and one was nil.
In that case, I would like to ignore it and average the 29 other
values.  Ideally, the output would be annotated in a way that 
indicated that it was an average of 29 values.  If one of
the values was NaN, say because of a divide by zero bug,
I'd like to know about it and not ignore it silently.

One solution would be to have taking the value of a nil
token with intValue(), doubleValue() etc. throw an exception.
However, this seems slow and somehow wrong.

Anyway . . .


I modified the code so that we now get an exception:

 >> cast([int],nil)
 Error invoking function public static ptolemy.data.Token
 ptolemy.data.expr.UtilityFunctions.cast(ptolemy.data.Token,ptolemy.data.Token)
 throws ptolemy.kernel.util.IllegalActionException

 Because:
 Conversion is not supported from ptolemy.data.Token$1 'nil' to the
 type [int].



We get similar exceptions for the other types as well.

I also modified ComplexToken so that we now have a ComplexToken.NIL
variable.  Each of the types has a NIL variable so we can preserve
type safety.

This led me to modify the IntMatrixToken.convert(ScalarToken, int) and
LongMatrixToken(ScalarToken, int) methods so they now also properly
barf if ScalarToken is nil.

These changes should probably go into the Kepler beta branch.

Is the plan to fold in the latest and greatest version of the
Ptolemy tree, or is the Ptolemy tree that will ship with Kepler
1.0 frozen?

If the Ptolemy tree in Kepler 1.0 is frozen, then these changes should
be folded in so as to prevent problems with nil tokens.  Note that we
need to be careful about Steve's array changes.  I'm not shipping
Steve's array changes in 6.0.1, which is due out sometime this month.
I think the change to data/type/MatrixType.java was the only
one that involved Steve's array changes.  When merging these
changes in, it would be worth doing a diff.

2007-01-03 10:46  cxh

	* ptolemy/data/: IntMatrixToken.java
	(1.111), LongMatrixToken.java (1.79), test/NilToken.tcl (1.10):
	Added tests for convert() methods.  IntMatrixToken and
	LongMatrixToken convert(ScalarToken, int) now properly handle nil
	Tokens

2007-01-03 09:39  cxh

	* ptolemy/data/type/MatrixType.java (1.7): Calling convert(nil) now
	throws an exception

2007-01-03 09:39  cxh

	* ptolemy/data/expr/test/Functions.tcl (1.39): Added tests for the
	cast expression language function, especially for casting nils.

2007-01-03 08:22  cxh

	* ptolemy/data/ComplexToken.java (1.89): toString() on a nil token
	now returns nil


With regard to when to merge changes in to the Kepler release branch, it might
be worth updating it sometime soon as Edward's performance
enhancements could help Kepler users.  I would merge from the Ptolemy
beta branch instead of merging from the Ptolemy head.

I'm getting fairly close to shipping 6.0.1 from the beta branch,
I don't anticipate any significant new changes, only minor
bugs.

_Christopher