[kepler-dev] Proposal: Kepler should work off a snapshot or release of Ptolemy instead of the trunk of Ptolemy

[Christopher Brooks]

Hi David,
Sorry about the gt problem, it was my mistake.  Things
worked in my Kepler tree, I think the problem was the gt files
weren't really gone when I tested.  I also sent email to
Kepler-dev that stated that the changes went in and that
this was up for discussion.

You should definitely send email when these things come up.
There was some traffic about this on ptdevel, I'll make sure
that you are added to that list.

I spend a certain amount of time tracking down build system bugs. 
Usually I try to reproduce it in a clean tree and then send you mail.
Feel free to do the same and quickly send email as opposed to
spending debugging time.

I've also spend many days on Kepler in the last weeks.  Cleaning
up exp and attending the Kepler meetings were both interesting
and big time consumers.

Feel free to manage and support working from a snapshot.  It should
only take an hour or two to validate a snapshot.  A more comprehensive 
test suite would speed up validation of the snapshot. The downside is
added confusion about what version of what is necessary.
Personally, I'll be working from the head of both trees.  The downside
to working from a snapshot is that upgrading can be very difficult
sometimes.  We have a developer who works on the Ptolemy tree and he
does not regularly update.  When he does update, it takes a couple of
days of work.  I think it is less effort to work from the head and avoid
batch merges, but this is just me.

BTW - is there anyway that all the svn messages for branching can be
combined into one svn checkin?  I received dozens of messages yesterday.

_Christopher

Matt Jones wrote:
> David,
> 
> Thanks for the thoughtful note.  We have discussed this issue a number
> of times over the last few years. In general, results from previous
> discussions agreed with your conclusions that a stable development
> platform is important.  We even were working off of a ptolemy snapshot
> for the first two years or so, but ultimately switched to the trunk
> when we determined that the periodic sync activity with the ptolemy
> trunk was not happening, mainly due to lack of an identifiable person
> to follow through on the sync.  So, the difficult issue that has
> always prevented us from using snapshots is determining who is
> responsible for these roughly monthly updates.  While Kepler/CORE is
> funded, we can fund infrastructure work like this off of that.  But in
> a year or so when we only have external-project developers, can we be
> sure someone will sync the snapshot with the ptolemy trunk?  If not,
> we will steadily drift out of sync, and future improvements and fixes
> to ptolemy become that much more difficult to incorporate.  The longer
> the time between syncs, the harder it is to sync.  Nightly builds
> against the ptolemy trunk ensure that we continue to be able to sync.
> 
> One way to view this is that the new release process with its
> associated teams based on our new module system will create a stable
> development point for people who want or need that.  Once we start
> making real module releases with Kepler 2.0.0, anyone who wants
> uninterrupted development time without distractions from trunk changes
> would simply use a release as their development point.  They would be
> unaffected by changes in the trunk.  Other developers, such as those
> responsible for maintaining the kepler 'kernel' modules, would
> presumably work off of the trunk, syncing with ptolemy until any new
> changes are working properly, then release new versions of the kepler
> and ptolemy kernel modules.  These would then be the basis for the
> next full kepler release, at which point other extension developers
> would be able to leapfrog ahead and start developing against the newly
> released, and stable, version of kepler.  In this scenario, we do
> snapshot ptolemy as a module each time we want to release a new
> version of the kernels that depend on ptolemy. Maybe it should be the
> Build and Release team's job to propose when new releases of the
> kepler kernel modules and ptolemy module are needed?
> 
> This approach allows us to both have a stable set of modules for
> developers that minimizes disruptions and to keep the trunk relatively
> synchronized with ptolemy so that we don't again drift out of sync.
> 
> Matt
> 
> 
> On Thu, May 14, 2009 at 2:44 PM, David Welker <david.v.welker at gmail.com> wrote:
>> Hi all,
>>
>> I wanted to see how people felt, including people working on Ptolemy, about
>> the following proposal. Instead of having the trunk of Kepler run off the
>> trunk of Ptolemy, instead we should have it run off a snapshot or release of
>> Ptolemy. Then, according to a set schedule, Kepler could synchronize the
>> trunk of Kepler with the trunk of Ptolemy. This way, we can remain highly
>> coordinated with Ptolemy and the Ptolemy community, but not be impacted by
>> minor (yet important) changes.
>>
>> Overall, I am really impressed with how stable the trunk of Ptolemy is.
>> Compilation problems and problems running are usually resolved quickly.
>> However, sometimes, things are not so easy and it isn't anyone's fault.
>> There are just too many moving parts.
>>
>> I think today was a case in point. The build continues to evolve, as it
>> must. Modules that depend on the build, such as loader and module-manager,
>> also continue to evolve. Ptolemy continues to evolve. All of this evolution
>> is absolutely necessary. But, it also conflicts with another need we have,
>> and that is a stable foundation upon which developers can work.
>>
>> Here is the story of what happened today:
>>
>> Ptolemy decided to move some configuration files -- gtEffigyFactory.xml and
>> gtTableauFactory.xml -- from one location to another. Specifically, from
>> ptolemy/configs/gt/ to ptolemy/configs/. This strikes me as a perfectly
>> sensible change. But, it also was pretty expensive in terms of developer
>> time. Unfortunately, our configuration.xml file in Kepler referred to the
>> old location of these files instead of the new locations.
>>
>> It took me at least 3 hours to find this problem. Not least of which,
>> because these changes in Ptolemy corresponded with significant changes I
>> have been making to loader and module-manger, not to mention the build. When
>> so many things are changing at once, it is challenging to isolate an error
>> like this. I spent a significant amount of time looking for how the build
>> system or loader was somehow causing this error, when in fact it was a
>> change in ptolemy without a corresponding change to a configuration file
>> that was the real culprit.
>>
>> Of course, the amount of time it took me to find this error is only a small
>> part of the cost. I think this change happened sometime yesterday. I didn't
>> run into this error until I updated. It basically broke Tim's build
>> yesterday, and at some point he had to stop working. I know it also broke
>> Aaron's build too. So, we have what turns out to be a rather minor (yet
>> important) change sucking up multiple hours for multiple developers. It
>> seems pretty clear that the decision to synchronize the trunk of Kepler with
>> the trunk of Ptolemy in real time is pretty expensive. So, we really have to
>> scrutinize this decision.
>>
>> Imagine a world where, instead of working off the trunk of Ptolemy, we work
>> off a snapshot that is made on a regular basis, say once a month. When we
>> decide to migrate the trunk of Kepler to work with the trunk of Ptolemy on a
>> monthly basis, if we found that Kepler suddenly stopped working, we would
>> immediately know where the problem was. There must have been a change in
>> Ptolemy that made it incompatible with Kepler. So, this sort of problem
>> would take much less time to hunt down, simply out of virtue that the set of
>> potential causes would be much smaller. Secondly, and more significantly,
>> the costs of such incompatibilities could be absorbed by one expert who
>> would be well equipped to figure out how compatibility was breaking down,
>> rather than subset of the entire Kepler community that chooses to work off
>> the trunk of Kepler. Only when compatibility was ensured would a new
>> snapshot that reflected the latest trunk of Ptolemy be used.
>>
>> I don't really see any downsides to this proposal. We would still be working
>> in a very collaborative manner with Ptolemy. Only, the costs of such
>> collaboration would be much lower.
>>
>> Does anyone else have any thoughts on this proposal?
>>
>> David
>> _______________________________________________
>> Kepler-dev mailing list
>> Kepler-dev at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>
> 
> 
> 

-- 
Christopher Brooks (cxh at eecs berkeley edu) University of California
CHESS Executive Director                      US Mail: 337 Cory Hall
Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
ph: 510.643.9841 fax:510.642.2718	      (Office: 545Q Cory)
home: (F-Tu) 707.665.0131 (W-F) 510.655.5480