[kepler-dev] where do actor packages meant to work with Kepler 1.0 go in the repository?

[Timothy McPhillips]

Hi Chad,

I like your suggestions.  I've added some questions and comments of  
my own:


On Jun 26, 2008, at 9:18 AM, Chad Berkley wrote:

> Hi Tim,
>
> See my thoughts below.
>
> Timothy McPhillips wrote:
>> Dear All,
>> With the Kepler 1.0 release out and the migration to svn complete  
>> I am about to begin migrating into the Kepler repository some code  
>> we've been developing here at UCD.  Some of these checkins will be  
>> relatively straightforward, but other code will present some  
>> integration, source code management, and build system challenges  
>> I'd really like to get everyone's input and help with.
>> *** Upcoming checkins ***
>> 1.  First, I will be committing a new version of the COMAD  
>> framework to src/org/kepler/comad.  The current version in Kepler  
>> is included under src/org/nddp.  The major difference with the new  
>> version is that it includes comprehensive support for recording  
>> provenance from COMAD workflows and storing the results as  
>> provenance-annotated 'trace' files.  2.  Second, we would like to  
>> begin committing the code specific to the pPOD extension to Kepler  
>> (phylogenetics actors and data types for the AToL community).   
>> This integration will be trickier.  Please see "complications" below.
>> 3.  Third, when the COMAD and pPOD sources have been successfully  
>> integrated with the main Kepler system, I will delete the code  
>> under src/org/nddp, this package having been superseded by the two  
>> sets of checkins above.
>> *** Complications  ***
>> Ok, here's the conundrum.  The pPOD extension is mostly a set of  
>> actors for phylogenetics analyses.  Prior to the Kepler 1.0  
>> release we provided a preview release <http:// 
>> daks.genomecenter.ucdavis.edu/kepler-ppod/> of the pPOD actors  
>> (based on the COMAD framework) to the AToL <http://atol.sdsc.edu/>  
>> community (see a poster <http://daks.genomecenter.ucdavis.edu/ 
>> kepler-ppod/poster.pdf> and presentation <http:// 
>> daks.genomecenter.ucdavis.edu/kepler-ppod/Kepler-Ppod.pdf> about  
>> the Kepler/ppod preview).  This was not meant for production use,  
>> but rather as a way to get feedback from the community on our  
>> work.  However, in the very near future we would like to release  
>> this code as a package of actors and community-specific  
>> customizations to the Kepler GUI.   Note that this release will  
>> not be targeted only at an abstract community of folks who might  
>> possibly try it out, but also at particular scientists who have  
>> asked for specific workflows based on the pPOD package and who  
>> need them very badly for their own research.)
>> As you might expect, we do *not* want to:
>> N1.  Wait for the next release of Kepler before distributing these  
>> actors. N2.  Release a package of actors that has been tested only  
>> against  the revision of Kepler that we happen to find at the  
>> trunk of the Kepler repository on a particular day. Instead, we  
>> want to (read 'must') release a package of actors that works with  
>> Kepler 1.0.  When another release of Kepler is made (Kepler 1.1,  
>> say), we will need to release a version of the pPOD actors that  
>> works with *that* version of Kepler.  There are number of  
>> questions implied by this need to release a package of actors  
>> designed to work with a particular, supported release of Kepler:
>> Q1. How do we build the pPOD actors against Kepler 1.0 rather than  
>> against the current revision in the repository?
>
> One way to do this would be to have a build.xml file in your ppod  
> source tree that just knows how to build ppod.  The build.xml file  
> for kepler could be made to import the ppod build file when ppod  
> needs to be built.  In the future, we could redesign the kepler  
> build.xml file to allow importing different build files from the  
> command line so that the build file itself does not need to be  
> altered.

An extension-specific build file in each extension source tree sounds  
like a good idea, as long as it doesn't require duplicating  
functionality provided by the central build.xml file.  There is more  
to do than build the Java sources, too.  Actor package extensions  
will have their own set of kar source files that will be need to be  
jarred up the same way kar sources for actors in the core of Kepler  
are, for example.  The build system ought to use the central support  
for building kar files rather requiring that package builders  
reinvent or copy this functionality.  (The hacked version of the  
build file we use at UCD deals with this issue--more or less).

>
>> Q2.  Where should we put the pPOD actors and support classes in  
>> the Kepler repository directory structure?  If we put them in,  
>> say, src/org/ppod (under kepler), then all of this code will be  
>> seen by the build system and all kepler developers will find  
>> themselves building code that is not necessarily meant to be  
>> compiled against the current revision of Kepler in the  
>> repository.  If any classes the pPOD package depends on elsewhere  
>> in Kepler are renamed, moved, removed, or changed significantly,  
>> the pPOD code will no longer build against the current revision of  
>> Kepler in the repository.  The build will break.
>
> if the build file system is used, the ppod actors would not be  
> built unless the user told the build system to build them.

So there would be a place in the repository for source code trees  
that do not get compiled by default with the core of Kepler?  Excellent.

>
>> A further complication is that once the 1.0-compatible version of  
>> the pPOD package is out, we likely will want to begin building  
>> against the latest revision of Kepler in the repository until the  
>> next version of Kepler is out, at which point we'll want to build  
>> against *that* supported version of Kepler.  In short, we need to  
>> be able to choose which revision of Kepler in the repository we  
>> build the pPOD package against, without disturbing folks working  
>> with the current revision of Kepler.
>
> I think this could be done with ant.  you'll just need to have your  
> build file make sure that the current checkout of kepler is the tag/ 
> branch that you want.
>

Cool.

>> *** Solutions? ***
>> S1.  On the surface this problem sounds like one that could be  
>> solved with branches.  And I believe it possibly could, and that  
>> branches likely will be involved in some way (for example, we will  
>> still want to provide bug fixes to the 1.0-compatible version of  
>> the pPOD actors when we're working on the 1.1-compatible  
>> release).  However, is this how we want to treat the Kepler 1.0  
>> release branch in the repository?  Do we want to be checking  
>> hundreds of source files into the Kepler 1.0 release branch  
>> indefinitely, and almost certainly introducing new bugs and  
>> instabilities there?
>
> Kepler 1.0 is released, so we can no longer make changes to that  
> branch and still call it 1.0.  This is not an option, imho.

I agree with this.

>
>> S2.  An alternative would be to create a separate source tree for  
>> pPOD in the svn repository (as a peer to kepler, for example).   
>> The build system would not include this 'extension' when compiling  
>> the Kepler java sources.  Instead, we might build the pPOD package  
>> against the Kepler 1.0 release jar itself.  This approach might  
>> scale better and enable everyone providing their users 1.0- 
>> compatible actors to share their work with the community through  
>> the Kepler repository without requiring that their code work with  
>> the current revision of Kepler.
>
> I kind of think of each directory in the repository as its own  
> repository anyway.  I think the compilation/non-compilation of a  
> given directory should be controlled by the build file(s).  I don't  
> think its necessary to have an entirely new repository for  
> extensions.  If this is something that is needed by the greater  
> community, we could create a high level directory in the kepler  
> repository for extensions (kepler/extension) and have a  
> hierarchical build system built around that.

I agree a new repository is not needed.  A directory for hosting  
extensions would be great.  I'd start checking in the COMAD and pPOD  
extensions today if there were such a place in the repository!

>
>> The problem with this second approach is that it might not be so  
>> easy to build against the current revision of Kepler, should one  
>> choose to do so (and as we likely will between releases of the  
>> pPOD package).  At least, not without building the Kepler jar each  
>> time.
>
> I think the build file in the extensions directory could easily  
> build kepler to the version you need and put the jar file into the  
> extensions dir for use.  then it wouldn't need to be built multiple  
> times and you'd guarantee that you were building against the  
> version of kepler you want.

Sounds good!

>
>> S3.  A variant of S2 would be to enable developers to tell the  
>> build system to optionally include extensions in the build.  With  
>> the extensions stored as peers to the kepler directory in the  
>> repository, developers could choose which extensions to check out,  
>> and specify different branches or tags for Kepler and for each  
>> extension checked out.
>
> I guess my approach above is similar to this except that it uses  
> the main kepler repository instead of different repositories.

Yep, I don't see a need for different repositories for each  
extension.  On the other hand, it would be nice if the build system  
could build extensions not stored in the central repository, too  
(e.g., extensions stored in other repositories but that have been  
checked out out to a developer's machine).

Tristan--do you have thoughts about this?

>
> I'm kind of thinking off the cuff here so I hope my ideas don't  
> seem too malformed.  Maybe we could have a call next week to chat  
> about this.
>

Let's definitely chat soon.  It'd be great to move quickly on this.

All--who wants to participate in that discussion?  Tristan?   
Christopher Tuot?


Thanks, Chad!

Tim