[kepler-dev] NIL token
Christopher Brooks
cxh at eecs.berkeley.edu
Wed Apr 5 10:18:17 PDT 2006
Hi Tristan,
Sorry about the problems integrating the latest Ptolemy head.
Haiyang Zheng has been doing some rearranging. It would be good
to get rid of the Kepler specific version of Director, this
problem will only get worse. There was a bug filed for
moving more of this code out of exp, but I think it is of lower
priority than other tasks. I got bit by something similar, I had
forgotten to update the Kepler version of TypeLattice.
You wrote:
> You mentioned in a previous email that you used to have nil-ness as
> a property of the token rather than a state like the current
> implementation. What were the problems with this system?
When we reviewed the code, Edward suggested a NIL instance of
the token instead of using a method to set a flag.
The advantage of using a NIL instance is that we don't add a boolean
to each Token. Another advantage is that to check if a token is
nil, we check to see if it is equal to NIL instead of checking
a boolean, which is probably faster.
One disadvantage is that the XXXToken(String) constructor can no
longer be used to create a nil token because the constructor can't
return a different instance. We'd like something like:
public FooToken(String init) {
if (init.equals("nil") {
return Token.NIL;
}
}
but that won't work because the constructor can't return a different
instance. Thus, the String constructors don't support constructing
nil tokens.
Also, each Token needs to have its own NIL because the convert()
method returns a more specific type than the type of the argument, so
we have to have IntToken.NIL instead of Token.NIL:
public static IntToken convert(Token token) throws IllegalActionException {
...
if (token == null || token.isNil()) {
return IntToken.NIL;
}
Also, I had to introduce a type called niltype, (to differentiate it
from nil). Token.NIL is of the niltype, which is losslessly
convertible to UnsignedByteToken, BooleanToken, IntToken, LongToken
and DoubleToken. The reason this is safe is because any operation
that is performed on a NIL token usually results in a NIL. Some
operations, such as average, could choose to ignore the NIL token.
In general, if you have a NIL, it can be seen as being of any type and
thus it can be losslessly converted in to other types that have
NILs.
One other restriction is that the MatrixTokens do not support nil
elements. The reason is that MatrixTokens are to be used for places
where speed is needed. The underlying element type in a MatrixToken
does not have support for nil types. For example IntMatrixToken has
an int[][] matrix behind it. To support nil tokens here, we would
need to add sparse matrix that would note which elements are nil.
This could be done, but it points to creating classes that extend
XXXMatrixToken rather than modifying the base XXXMatrixToken and
getting a speed hit.
Another issue is that NIL StringTokens have the toString() value
"nil", yet constructing a StringToken with new StringToken("nil")
results in a StringToken with three letters, n, i and l, which is
not a NIL token. Also, for historical reasons, new StringToken(null)
returns a StringToken of length 0, aka the empty string.
Currently, new StringToken(null) could not return StringToken.NIL
anyway, but it seems odd.
At this point, I'm tempted to leave the ArrayTokens as they are
with support for zero length arrays and for ArrayTokens that have
elements that are nil and move on.
_Christopher
--------
Hi Christopher,
After a lot of messing around to get kepler to compile over the latest
ptolemy cvs (in the end i had to remove the ClusterW stuff and the
Director.java file from $KEPLER/src/exp/ directory, as well as add a few
methods to FSMActor.java in ptolemy)... anyways, my test workflow worked
exactly how i wanted it to work. so it looks like you've solved the 0
length array problem for now :).
I am on the same level of confusion as you with the difference between
an empty arraytoken and a nil arraytoken. From my understanding of the
definition of a nil, a nil arraytoken would specify an array should (or
could) exist, but is currently missing. But it may not be true to say
that a zero size array is missing, since it's not really missing, it
just has a size of zero.
Currently when i check for a zero size arraytoken i check the length()
of the array rather than it's nil-ness, since that's what i'm looking
for, and that works perfectly for me. I'm still concerned about the
token being considered nil, but it wont keep me up at night :).
I think it comes into the same league as the stringtoken making nulls
into an empty string, the difference is currently so subtle that it will
probably never effect anyone.
It may though, in the future, so maybe it is worth the time to try and
specifically define the differences before the code is so deeply
embedded that changing it would cause major issues.
You mentioned in a previous email that you used to have nil-ness as a
property of the token rather than a state like the current
implementation. What were the problems with this system?
Thanks
--Tristan
Christopher Brooks wrote:
> Hi Tristan,
> I just checked in a change to ptolemy.data.type.TypeLattice so that
> the niltype is convertible to an UnsignedByte. Yesterday I checked
> in changes that support nil in UnsignedByte and empty ArrayTokens.
>
> I'm not so sure I understand the difference between and empty
> ArrayToken and a nil ArrayToken, so there could be problems there.
> Perhaps it is a little like the difference betweeen an empty
> StringToken a nil StringToken.
>
> $PTII/ptolemy/actor/lib/test/auto/NilTokenTypeTest.xml
> contains the simplistic type tests that I'm using to verify these type
> changes. The model consists of Const actors connected to Test actors,
> where the Consts have the value nil and the type of the port is set
> to int, double, long, or unsignedByte. The unsignedByte test was
> failing until my most recent changes to TypeLattice. I also added
> tests that test empty arrays in a similar fashion.
>
> If, after my most recent change to TypeLattice, empty arrays are still
> not working for you, please send me a really simple model or code
> fragment so I can construct a test and fix it. I suspect that
> my most recent change will do the trick, but it could be I missed the
> bug that is affecting your model and fixed a different bug.
>
> I spent some time yesterday adding tests to the testsuite for nil
> tokens and discovered some bugs, like that we need an UnsignedByte nil
> token. However, more tests are needed. This work will probably not
> happen until tomorrow.
>
> _Christopher
>
>
>
>
> --------
>
> Hi Christoper,
>
> I understand the need for the nil token now, thanks for clearing that
up.
>
> I take it you're still in the middle of getting the ArrayToken nils t
o
> work? I just want to check to make sure I'm not doing something wrong
or
> there's a bug.
>
> I am still getting a conversion error when trying to send a 0 element
array
> .
>
> ptolemy.kernel.util.IllegalActionException: Run-time type checking
> failed. Token {} with type {nil} is incompatible with port type:
> {unsignedByte}
> in .<Unnamed Object>.Binary File Reader (dart).output
> at ptolemy.actor.TypedIOPort._checkType(TypedIOPort.java:741)
> at ptolemy.actor.TypedIOPort.send(TypedIOPort.java:463)
> at org.dart.BinaryFileReader.fire(BinaryFileReader.java:117)
> at ptolemy.actor.AtomicActor.iterate(AtomicActor.java:309)
> at
> ptolemy.actor.sched.StaticSchedulingDirector.fire(StaticSchedulingDir
ector.
> java:170)
> at ptolemy.actor.CompositeActor.fire(CompositeActor.java:331)
> at ptolemy.actor.Manager.iterate(Manager.java:613)
> at ptolemy.actor.Manager.execute(Manager.java:322)
> at ptolemy.actor.Manager.run(Manager.java:987)
> at ptolemy.actor.Manager$3.run(Manager.java:1028)
>
> the port is assigned a type via the following code:
> output.setTypeEquals(new ArrayType(BaseType.UNSIGNED_BYTE));
>
> Thanks
> --Tristan
>
> Christopher Brooks wrote:
> > Hi Tristan,
> > I think I have most of the implementation of an empty array working
.
> >
> > I agree that having new StringToken(null) create an empty string is
> > probably wrong. We could change it, but I'm not sure if it would b
uy
> > us much, and it might break things.
> >
> > The way I'm thinking of the nil token is that it represents missing
> > data that could have any value. Certain operations work on nil
> > values, for example any value added to a nil returns a nil. Howeve
r,
> > operations like average should return regular numeric data, even if
> > one or more elements of the average are nil. However, we do get
> > increased uncertainty in the average, and we don't have any way
> > of showing that.
> >
> > I took on the nil token task because it was at the top of the bug l
ist
> > for Kepler and it seemed like it needed to be implemented by someon
e
> > who would make changes to the Ptolemy tree. I somewhat understand
why
> > we need to represent missing data, but I make no claims as to
> > whether this is the right implementation. I'm sure that some peopl
e
> > wish Java had better support for missing data.
> >
> > The relationship between nil and null is subtle.
> >
> > Anyway, thanks for your comments, I may have more to say later.
> >
> > _Christopher
> >
> >
> >
> > --------
> >
> > Ok this is a bit of a rant, and since i probably don't know eve
n 5% o
> f
> > the internal workings of ptolemy/kepler i may be misguided in a
lot o
> f
> > this rant so please correct me anywhere i need correction :).
> >
> > Maybe we need to take a step back and look at what a Token actu
ally i
> s,
> > and what we're actually trying to achieve from a nil token. Fro
m the
> > javadoc in the Token class a Token is simply a container for st
andard
>
> > java objects and native types.
> >
> > So, What are we trying to achieve with a nil token? There must
be a v
> ery
> > good rationale to why we need such a mechanism, but i can't for
the l
> ife
> > of me figure out what it is. Are we trying to signify the lack
of a
> > variable, or a variable that has a null value?
> >
> > If we are talking about the lack of a variable, i.e an empty co
ntaine
> r
> > then for one, I think nil is the wrong syntax for this.
> >
> > Now, should an empty container be allowed? Sure. But only for u
se as
> a
> > trigger (i.e. the EventToken). I don't think it should be used
to
> > signify the lack of a value which may or may not exist.
> >
> > If we are trying to say the value in the container is null. It
just
> > doesn't make sense.
> >
> > If we look at the native types like ints doubles booleans etc,
they
> > cannot be null. They either exist, or they don't. So if the exp
ressio
> n
> > "int i = null;" is invalid in java, how can we say "IntToken i
=
> > Token.NIL;" (or IntToken.NIL)? I think forcing a specific condi
tion t
> o
> > be handled as null can limit the programmer. Especially if they
need
> to
> > use the value Integer.MAX_VALUE and not have other actors view
it as
> null.
> >
> > So what about Objects that can be assigned to null?
> >
> > What's wrong with something like:
> > Object o = null;
> > ObjectToken ot = new ObjectToken(o);
> >
> > or
> > Object[] o = null;
> > ArrayToken at = new ArrayToken(o);
> >
> > Sure, the null values stored in the Tokens may cause exceptions
, but
> i
> > think those exceptions should be handled by the programmer rath
er tha
> n
> > avoided. I don't see why the container itself needs to know any
thing
> > about the object it contains (except for the type).
> >
> > I feel the current direction is moving more away from the Token
s bein
> g
> > transport containers to actually being the variables themselves
.
> >
> > For example the StringToken container converts Strings which ar
e null
> to
> > Strings which have a value "". This doesn't fit with the defi
nition
> of
> > a container. It's not just transporting the objects, it's manip
ulatin
> g
> > them as well. There is a very distinct difference between a Str
ing se
> t
> > to null and "", this conversion adds ambiguity to the String ob
ject.
> >
> > Another example is the ArrayToken container. When the Token is
though
> t
> > of as just being a container for arrays it makes no sense that
empty
> > arrays cannot be contained inside. Any array that is valid in j
ava
> > should be containable inside the token.
> >
> > As with my problem, I'm not trying to alert the other actors of
the l
> ack
> > of a token, I want to send them an array that has no values. In
java
> > there is a distinct difference between: "Object[] o = null;" an
d
> > "Object[] o = Object[0];". The ArrayToken container should just
> > transport my array object and not be concerned about how long i
t is e
> tc.
> >
> > What about the type of the array? maybe it should be specified
by the
>
> > programmer (the same way that ports have to be set), rather tha
n bein
> g
> > interpreted from the elements in the array, and have the base c
lass a
> s
> > BaseType.OBJECT incase no type is specifically assigned. i've n
ot put
> a
> > lot of thought into this just yet, i'm sure i'll be doing a lot
over
> the
> > weekend :)
> >
> > In conclusion, 1. Scrap the nil token idea, and think more of h
ow you
>
> > would represent the missing values in a normal java program, an
d how
> you
> > could contain that representation in a Token. And 2. the ArrayT
oken (
> and
> > possibly some of the other tokens) need to be reworked so they
act mo
> re
> > like containers.
> >
> > Anyone else have any thoughts (anything to add, or critisisms)
about
> the
> > things i've brought up?
> >
> > --Tristan
> >
> > Edward A. Lee wrote:
> > > At 11:53 PM 3/29/2006, Christopher Brooks wrote:
> > >> I just checked in some changes so that some of the Tokens ha
ve the
> ir
> > >> own NIL
> > >> token. Token.NIL, DoubleToken.NIL, IntToken.NIL, LongToken.N
IL and
> all
> > >> of these NILs are of a new type, "NilType". NilType is loss
lessly
> > >> convertible to (from?) DoubleType, IntType, LongType etc.
> > >>
> > >> I think we need these Token specific NILs so that the conver
t()
> > >> method will work.
> > >
> > > This looks like the right solution to me...
> > >
> > > An awkwardness is that every token class has to have a NIL
> > > and an appropriate convert method, but this awkwardness is
> > > only visible to token class developers.
> > >
> > > I think that having an empty array be ArrayToken.NIL
> > > might be a very good solution too... We could then augment
> > > the expression language so that the string "{}" returns
> > > an instance of this. It's element type is NilType.
> > > Since this is losslessly convertible to every type
> > > (including ArrayToken, by the way), then I think it should
> > > just work!
> > >
> > > Amazing... There may be a type system paper in this...
> > >
> > > Edward
> > >
> > >
> > >
> > > ------------
> > > Edward A. Lee
> > > Professor, Chair of the EE Division, Associate Chair of EECS
> > > 231 Cory Hall, UC Berkeley, Berkeley, CA 94720
> > > phone: 510-642-0253 or 510-642-0455, fax: 510-642-2845
> > > eal at eecs.Berkeley.EDU, http://ptolemy.eecs.berkeley.edu/~eal
> > >
> >
> > --
> > Tristan King | Ph: (07) 4781 6911
> > DART project team | Email: Tristan.King at j
cu.edu
> .au
> > James Cook University | Web: http://dart.edu.
au
> > Townsville QLD 4814 | http://plone.jcu.edu.
au/dar
> t/
> > Australia |
> > --------
> >
>
> --
> Tristan King | Ph: (07) 4781 6911
> DART project team | Email: Tristan.King at jcu.edu
.au
> James Cook University | Web: http://dart.edu.au
> Townsville QLD 4814 | http://plone.jcu.edu.au/dar
t/
> Australia |
> --------
>
--
Tristan King | Ph: (07) 4781 6911
DART project team | Email: Tristan.King at jcu.edu.au
James Cook University | Web: http://dart.edu.au
Townsville QLD 4814 | http://plone.jcu.edu.au/dart/
Australia |
--------
More information about the Kepler-dev
mailing list