[kepler-dev] how to distribute, upgrade, and support Ptolemy under 2.0?

[David Welker]

I thought I would chime into this discussion.

(1)
It is correct that some people will want to work at the head of  
ptolemy or the head of another module. However, such people will be  
developers who will be prepared to deal with the issues that arise  
from dealing with an unstable and constantly changing environment. The  
Module Manager GUI is primarily for scientists or developers who want  
to work with modules that are stable. Hence the concept of  
"publishing." When you publish something, you are saying that there  
will be no further changes, with the exception of patches which are  
supposed to consist of bug fixes and always be backwards compatible.

If someone wants to work from the head of one or more modules, they  
will use the build system, not the Module Manager GUI.  The Module  
Manager GUI only works with published modules. Published modules must  
always come from branches, and not from the head, to enable sensible  
patching. It must be emphasized that publishing and patching must  
occur from a branch, and never from the head. The reason for this is  
to ensure that for any published module with a particular version  
number, there is source code in a repository that roughly corresponds  
to that published modules, and from which patches can be made. If one  
were allowed to publish from the head, the problem is that the head is  
likely to vary radically over time. In effect, a patch to version  
1.A.B and version 3.C.D at some point would both have to come from the  
head, which would render 1.A.B+1 and 3.C.D+1 identical. Basically, the  
you need a branch for every module so you have a place to put unique  
patches. It cannot all be done from the head.

(2)
The issue of referring to other repositories is an interesting one.  
There are two different types of locations to consider: (1) The  
locations of branches from which publishing and patching can occur and  
(2) the locations of published modules. Right now, the list of  
locations consists of one location for branches and one location for  
published modules. This, in principle could be expanded...

Before we do so, we should consider the following principle. The more  
locations or nodes that contain just part of what is needed, the less  
robust the solution from a network standpoint.

Here is an example: Assume that there is one node that has everything  
that you need, and the chance of successfully accessing that node is  
95%. Overall, you would have a 95% chance of getting everything you  
need. Now, assume you divide the parts that you need across multiple  
nodes (with no replication). The following table illustrates how  
reliability (i.e. the probability of success) decreases as the number  
of nodes you depend on increases.

1: 95.0%
2: 90.3%
3: 85.7%
4: 81.4%
5: 77.4%
6: 73.5%
7: 69.8%
8: 66.3%
9: 63.0%
10: 59.9%

The point is that we need to think carefully about our strategy for  
allowing published modules to exist in multiple repositories. To the  
extent that we actually duplicate everything that is needed in  
multiple repositories, we increase reliability. To the extent that we  
keep different necessary parts in multiple repositories, we decrease  
reliability.

In my view, it would be fine to have branches in multiple  
repositories. But, I think it would be optimal if all published  
modules were in one repository to the extent feasible. This is  
especially so for all core modules, such as ptolemy.

(3) To the extent that we jar .class files, that does no imply that  
modules will no longer have source code. In fact, they will. The point  
of jarring is because it would be inappropriate for installed  
instances of Kepler to have to compile modules. However, modules will  
still have everything that exist when developers work with them. In  
any case, As far as GPL goes, if I remember correctly, I do not  
believe it is necessary to distribute the source code with every  
binary. What is required is that the source code be readily available.  
However, even if it were required for the source code to be  
distributed, this requirement would in fact be met. Finally, it would  
be inappropriate to distribute anything that is licensed GPL with the  
core of Kepler.







On Oct 6, 2009, at 9:43 AM, Christopher Brooks wrote:

> Hi Tim,
>
> Interesting issues.  It would probably be worth looking at how
> Eclipse extends OSGi to ship modules.  There might be some insight
> into some of these issues.
>
> My comments are below.
>
> Timothy McPhillips wrote:
>> Hi Christopher,
>> Thanks for your comments!  See below for clarifications and answers  
>> to your questions:
>> On Oct 2, 2009, at 12:34 PM, Christopher Brooks wrote:
>>> Hi Tim,
>>>
>>> The usage cases looks good, it helps to have something concrete
>>> to discuss.
>>>
>>> I'm slightly confused about your use of the term "publish".
>> Modules that are "published" will be visible to Kepler users via  
>> the Kepler module manager.  Published modules can be downloaded and  
>> installed from within Kepler.
>>>
>>> In my mind, a software product is made available in a
>>> prerelease form (alpha, beta, release candidates) and then released.
>>>
>>> For example, Ptolemy II 8.0 is available via svn, so does this mean
>>> it is published?  Eventually, it will be released, which
>>> means the source tree evolves very slowly after that,
>>> with critical bug fixes only.  How does the module scheme support  
>>> the
>>> development of releases?
>> According to this proposal, Ptolemy II 8.0 being available via svn  
>> does not make it "published" in the Kepler sense.  There will have  
>> to be a kepler module that represents Ptolemy somehow.  When this  
>> module is published, the Ptolemy 8.0 system will effectively be  
>> published to Kepler users. My question has to with how this Kepler  
>> module might best represent Ptolemy 8.0.  One option that might be  
>> reasonable if Ptolemy doesn't change much would be to include a  
>> jarred version of Ptolemy in the module.
>
> Jarring up a subset of Ptolemy II .class files and sources is one  
> possibility.
>
> Ptolemy II is published with source code, it would be nice if
> the module system included easy access to the Ptolemy II source code.
> For GPL'd software, you will need to make source code easily  
> available.
>
>
>>>
>>> When a module is finalized, then it will be using a particular
>>> revision or mostly unchanging branch of the kepler and ptII
>>> repositories.
>>>
>> Well, from an architecture point of view there won't be a  
>> monolitihic Kepler base system to which users can add optional  
>> modules that have been published, nor will published modules be  
>> able to declare dependency on, say, the entirety of one revision of  
>> the Kepler svn repository.  'Kepler' itself will be distributed, in  
>> its entirety, as a set of published modules.    It's going to be  
>> turtles all the way down.
>
> Will you publish modules for the JVM, the OS and the microcode on
> the CPU?  Just kidding.
>
> I like the idea of a system using itself all the way.
> However, you may want to consider providing a base system that is
> not optional.  Also, you may want to consider providing an interface
> and then implementations, much like how Eclipse works with OSGi.
>
> The module system should be able to refer to a revision in an external
> repository.  The module system should also be able to refer to
> the head of the remote repository.
>
>
>>> So, I guess where I'm confused is that there is no notion in the
>>> use scenarios that describe how often version numbers are changed?
>>>
>> This will be up to the maintainers of the various modules.  Module  
>> authors will be free to provide new versions of their modules as  
>> often as they please, and at different rates, as long as new  
>> versions depend exclusively on other modules that have been  
>> previously published.
>
>
> I think the notion of publishing a module is missing the notion
> of iterative software development where software is successively
> refined and then considered "done" and development moves on.
>
> A notion of stability, such as modules that have tests or have
> been inspected might be useful.  Ptolemy II has the red/yellow/green
> ratings, where red is likely to change, yellow has been design
> reviewed and green has been code reviewed.  It would be nice
> if the module system had some measure of quality.
>
>>> Maybe I'm missing it, but is there an obvious module that is the
>>> current development head that is working with the heads of the
>>> various repositories?
>>>
>>> I'm somewhat opposed to having lots of branches added to the ptII
>>> tree because it adds clutter.  Every module that uses Ptolemy II
>>> should not need a branch.
>>>
>> There need be only one branch of the Ptolemy module for each  
>> version of Ptolemy one wishes to provide patches (i.e., bug fixes)  
>> for.  To be clear, when we talk about creating branches for  
>> modules, we are not talking about branching the entire repository  
>> each time a minor or major version of any module is published.   
>> Rather, each module will have one branch--a branch of just itself-- 
>> for each major and minor version of that module that has been  
>> published.  Modules are branched and versioned independently.
>
> I don't understand the above.
> The way svn works is that one tags a tree and a branch is created.
> (Mercifully, I don't believe the entire tree is duplicated on the  
> server.)
> Having modules be branched and versioned independently is confusing
>
> Say I have package Foo-1.2.3 and it requires a particular revision of
> the ptII repository?  How do I do that?  Jarring up a portion of the
> ptII tree .class files is one solution, but having jars for every  
> revision
> is a bit of a waste (however, disk is cheap . . .).
> The source code for that jar should also be either jar'd up or  
> otherwise
> made easily available.  Hopefully, the module system does not need
> any modification (such as a branch) to the ptII svn tree, much like  
> the
> module system should not need to modify the JVM sources.
>
> The module system could be seen as a read-only client of some source  
> code
> providers such as Ptolemy or the JVM etc.  Perhaps this is a use case?
>
>
>
>
>>> It would be better to either require a particular revision
>>> of the ptII repository or else require a stable version, such
>>> as the 7.0 or 8.0.
>>>
>> If there will be no bug fixes to Ptolemy 8.0 then it need be  
>> published only once.  However, I do expect that we will want to  
>> roll out patches to Kepler that will require changes to Ptolemy.   
>> Each such set of change to Ptolemy that we want users of Kepler in  
>> the field right away will need to be published. My guess is that  
>> waiting for the 9.0 release of Ptolemy, say, might not be  
>> practical; I think depending directly or indirectly on an svn  
>> revision of Ptolemy that has not been released officially also  
>> would be problematic.
>
> The Ptolemy way is that if people want recent changes, they should use
> the head of the repository.  If they can't wait, then we sometimes
> roll out an informal stability point.  To do this requires looking
> at the test case failures and fixing as many as possible.  It also  
> requires
> verifying that the build is correct and other changes.
>
> Working from the previous stable release and fixing bugs is easier
> than working from the head because there is less work in the former
> than the latter.  However, for large changes, then it is easier to  
> work
> from the head.  For me, the dividing line is: bug fixes can go into
> the previous release as patches for awhile, usually 90 days, after  
> that, I focus
> on the head.  Enhancements almost always go into the trunk at the  
> head.
>
> So, it depends on if Kepler has bug fixes that need to go into the
> ptII tree or enhancements.
>
> I still don't understand why referring to a particular revision of
> the ptII repository would be problematic for the module system.
>
>>> Is it possible to later create a branch from a particular revision?
>>> For example if a module uses r50000 and now the tree is at r60000,
>>> can I go back and create a branch based on r50000 and make changes
>>> that are based on the r50000 tree?
>>>
>> In this proposal, any published major or minor version of a module  
>> can be branched to produce a new major or minor version of the  
>> module.  And any version that has been published can be patched at  
>> a later time and republished because the branches corresponding to  
>> each published version will be in the repository.
>
> Sorry about that, my question was more about whether it was possible  
> to
> create a branch after the fact in svn.
>
> After looking at the above, below is a summary:
> 1) How does the module system deal with external svn repositories?
> It sounds like you would prefer to tar up the .class files and  
> resources.
> What about source?
>
> 2) How will the module system be used to develop against the head
> of several repositories?  In otherwords, can I specify to use
> the head of ptII and other repositories?
>
> 3) Where is the notion of quality and stability in the module system?
> How do I know that one set of modules is more stable than another?
>
> What do other systems do?
>
> Anyway, interesting issues these . . .
>
> _Christopher
>
>
>
>
>
>
>
>>> Hope this helps,
>>>
>>> _Christopher
>> Is this clearer?  Thank you!
>> Tim
>>>
>>> Timothy McPhillips wrote:
>>>> Hi Christopher,
>>>> A while back I posted a proposal for how modules will be  
>>>> published, how new versions of modules will be distributed and  
>>>> installed, and how patches with bug-fixes to modules will be  
>>>> applied in Kepler 2.0:  https://kepler-project.org/developers/teams/build/documentation/proposed-usage-scenarios-for-publishing-and-patching-kepler-modules
>>>> One aspect of the approach described above is that branches of  
>>>> modules are maintained for each major and minor branch of  
>>>> published modules, so that patches can be created for each such  
>>>> version in the future.   What do you think about how Ptolemy  
>>>> could best fit into this scheme? Thanks for your thoughts!
>>>> Cheers,
>>>> Tim
>
> -- 
> Christopher Brooks (cxh at eecs berkeley edu) University of California
> CHESS Executive Director                      US Mail: 337 Cory Hall
> Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
> ph: 510.643.9841 fax:510.642.2718	      (Office: 545Q Cory)
> home: (F-Tu) 707.665.0131 (W-F) 510.655.5480
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev