[kepler-dev] how to distribute, upgrade, and support Ptolemy under 2.0?

[Christopher Brooks]

Hi Tim,

Interesting issues.  It would probably be worth looking at how
Eclipse extends OSGi to ship modules.  There might be some insight
into some of these issues.

My comments are below.

Timothy McPhillips wrote:
> Hi Christopher,
> 
> Thanks for your comments!  See below for clarifications and answers to 
> your questions:
> 
> 
> On Oct 2, 2009, at 12:34 PM, Christopher Brooks wrote:
> 
>> Hi Tim,
>>
>> The usage cases looks good, it helps to have something concrete
>> to discuss.
>>
>> I'm slightly confused about your use of the term "publish".
> 
> Modules that are "published" will be visible to Kepler users via the 
> Kepler module manager.  Published modules can be downloaded and 
> installed from within Kepler.
> 
>>
>> In my mind, a software product is made available in a
>> prerelease form (alpha, beta, release candidates) and then released.
>>
>> For example, Ptolemy II 8.0 is available via svn, so does this mean
>> it is published?  Eventually, it will be released, which
>> means the source tree evolves very slowly after that,
>> with critical bug fixes only.  How does the module scheme support the
>> development of releases?
> 
> According to this proposal, Ptolemy II 8.0 being available via svn does 
> not make it "published" in the Kepler sense.  There will have to be a 
> kepler module that represents Ptolemy somehow.  When this module is 
> published, the Ptolemy 8.0 system will effectively be published to 
> Kepler users. My question has to with how this Kepler module might best 
> represent Ptolemy 8.0.  One option that might be reasonable if Ptolemy 
> doesn't change much would be to include a jarred version of Ptolemy in 
> the module.

Jarring up a subset of Ptolemy II .class files and sources is one possibility.

Ptolemy II is published with source code, it would be nice if
the module system included easy access to the Ptolemy II source code.
For GPL'd software, you will need to make source code easily available.


>>
>> When a module is finalized, then it will be using a particular
>> revision or mostly unchanging branch of the kepler and ptII
>> repositories.
>>
> 
> Well, from an architecture point of view there won't be a monolitihic 
> Kepler base system to which users can add optional modules that have 
> been published, nor will published modules be able to declare dependency 
> on, say, the entirety of one revision of the Kepler svn repository. 
>  'Kepler' itself will be distributed, in its entirety, as a set of 
> published modules.    It's going to be turtles all the way down.

Will you publish modules for the JVM, the OS and the microcode on
the CPU?  Just kidding.

I like the idea of a system using itself all the way.
However, you may want to consider providing a base system that is
not optional.  Also, you may want to consider providing an interface
and then implementations, much like how Eclipse works with OSGi.

The module system should be able to refer to a revision in an external
repository.  The module system should also be able to refer to
the head of the remote repository.


> 
>> So, I guess where I'm confused is that there is no notion in the
>> use scenarios that describe how often version numbers are changed?
>>
> 
> This will be up to the maintainers of the various modules.  Module 
> authors will be free to provide new versions of their modules as often 
> as they please, and at different rates, as long as new versions depend 
> exclusively on other modules that have been previously published.


I think the notion of publishing a module is missing the notion
of iterative software development where software is successively
refined and then considered "done" and development moves on.

A notion of stability, such as modules that have tests or have
been inspected might be useful.  Ptolemy II has the red/yellow/green
ratings, where red is likely to change, yellow has been design
reviewed and green has been code reviewed.  It would be nice
if the module system had some measure of quality.

> 
>> Maybe I'm missing it, but is there an obvious module that is the
>> current development head that is working with the heads of the
>> various repositories?
>>
>> I'm somewhat opposed to having lots of branches added to the ptII
>> tree because it adds clutter.  Every module that uses Ptolemy II
>> should not need a branch.
>>
> 
> There need be only one branch of the Ptolemy module for each version of 
> Ptolemy one wishes to provide patches (i.e., bug fixes) for.  To be 
> clear, when we talk about creating branches for modules, we are not 
> talking about branching the entire repository each time a minor or major 
> version of any module is published.  Rather, each module will have one 
> branch--a branch of just itself--for each major and minor version of 
> that module that has been published.  Modules are branched and versioned 
> independently.

I don't understand the above.
The way svn works is that one tags a tree and a branch is created.
(Mercifully, I don't believe the entire tree is duplicated on the server.)
Having modules be branched and versioned independently is confusing

Say I have package Foo-1.2.3 and it requires a particular revision of
the ptII repository?  How do I do that?  Jarring up a portion of the
ptII tree .class files is one solution, but having jars for every revision
is a bit of a waste (however, disk is cheap . . .).
The source code for that jar should also be either jar'd up or otherwise
made easily available.  Hopefully, the module system does not need
any modification (such as a branch) to the ptII svn tree, much like the
module system should not need to modify the JVM sources.

The module system could be seen as a read-only client of some source code
providers such as Ptolemy or the JVM etc.  Perhaps this is a use case?




> 
>> It would be better to either require a particular revision
>> of the ptII repository or else require a stable version, such
>> as the 7.0 or 8.0.
>>
> 
> If there will be no bug fixes to Ptolemy 8.0 then it need be published 
> only once.  However, I do expect that we will want to roll out patches 
> to Kepler that will require changes to Ptolemy.  Each such set of change 
> to Ptolemy that we want users of Kepler in the field right away will 
> need to be published. My guess is that waiting for the 9.0 release of 
> Ptolemy, say, might not be practical; I think depending directly or 
> indirectly on an svn revision of Ptolemy that has not been released 
> officially also would be problematic.

The Ptolemy way is that if people want recent changes, they should use
the head of the repository.  If they can't wait, then we sometimes
roll out an informal stability point.  To do this requires looking
at the test case failures and fixing as many as possible.  It also requires
verifying that the build is correct and other changes.

Working from the previous stable release and fixing bugs is easier
than working from the head because there is less work in the former
than the latter.  However, for large changes, then it is easier to work
from the head.  For me, the dividing line is: bug fixes can go into
the previous release as patches for awhile, usually 90 days, after that, I focus
on the head.  Enhancements almost always go into the trunk at the head.

So, it depends on if Kepler has bug fixes that need to go into the
ptII tree or enhancements.

I still don't understand why referring to a particular revision of
the ptII repository would be problematic for the module system.

>> Is it possible to later create a branch from a particular revision?
>> For example if a module uses r50000 and now the tree is at r60000,
>> can I go back and create a branch based on r50000 and make changes
>> that are based on the r50000 tree?
>>
> 
> In this proposal, any published major or minor version of a module can 
> be branched to produce a new major or minor version of the module.  And 
> any version that has been published can be patched at a later time and 
> republished because the branches corresponding to each published version 
> will be in the repository.

Sorry about that, my question was more about whether it was possible to
create a branch after the fact in svn.

After looking at the above, below is a summary:
1) How does the module system deal with external svn repositories?
It sounds like you would prefer to tar up the .class files and resources.
What about source?

2) How will the module system be used to develop against the head
of several repositories?  In otherwords, can I specify to use
the head of ptII and other repositories?

3) Where is the notion of quality and stability in the module system?
How do I know that one set of modules is more stable than another?

What do other systems do?

Anyway, interesting issues these . . .

_Christopher







> 
>> Hope this helps,
>>
>> _Christopher
> 
> Is this clearer?  Thank you!
> 
> Tim
> 
> 
>>
>> Timothy McPhillips wrote:
>>> Hi Christopher,
>>> A while back I posted a proposal for how modules will be published, 
>>> how new versions of modules will be distributed and installed, and 
>>> how patches with bug-fixes to modules will be applied in Kepler 2.0: 
>>>  https://kepler-project.org/developers/teams/build/documentation/proposed-usage-scenarios-for-publishing-and-patching-kepler-modules
>>> One aspect of the approach described above is that branches of 
>>> modules are maintained for each major and minor branch of published 
>>> modules, so that patches can be created for each such version in the 
>>> future.   What do you think about how Ptolemy could best fit into 
>>> this scheme? Thanks for your thoughts!
>>> Cheers,
>>> Tim

-- 
Christopher Brooks (cxh at eecs berkeley edu) University of California
CHESS Executive Director                      US Mail: 337 Cory Hall
Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
ph: 510.643.9841 fax:510.642.2718	      (Office: 545Q Cory)
home: (F-Tu) 707.665.0131 (W-F) 510.655.5480