[kepler-dev] where do actor packages meant to work with Kepler 1.0 go in the repository?

[Chad Berkley]

I think we could make it general enough that you could build an 
extension out of a directory within kepler or from a different 
repository all together.  Like I said, I kind of view directories in the 
repository as different repositories anyway.

I definitely support making it so that functionality in the root 
build.xml file is not redundant.  I think ant does this pretty nicely. 
Our metacat product does something similar for the ecogrid extensions.

I'm available pretty much anytime next week monday - wednesday.  I'm 
busy most of Thursday and Friday is a holiday (Independence Day).

chad


Timothy McPhillips wrote:
> Hi Chad,
> 
> I like your suggestions.  I've added some questions and comments of my own:
> 
> 
> On Jun 26, 2008, at 9:18 AM, Chad Berkley wrote:
> 
>> Hi Tim,
>>
>> See my thoughts below.
>>
>> Timothy McPhillips wrote:
>>> Dear All,
>>> With the Kepler 1.0 release out and the migration to svn complete I 
>>> am about to begin migrating into the Kepler repository some code 
>>> we've been developing here at UCD.  Some of these checkins will be 
>>> relatively straightforward, but other code will present some 
>>> integration, source code management, and build system challenges I'd 
>>> really like to get everyone's input and help with.
>>> *** Upcoming checkins ***
>>> 1.  First, I will be committing a new version of the COMAD framework 
>>> to src/org/kepler/comad.  The current version in Kepler is included 
>>> under src/org/nddp.  The major difference with the new version is 
>>> that it includes comprehensive support for recording provenance from 
>>> COMAD workflows and storing the results as provenance-annotated 
>>> 'trace' files.  2.  Second, we would like to begin committing the 
>>> code specific to the pPOD extension to Kepler (phylogenetics actors 
>>> and data types for the AToL community).  This integration will be 
>>> trickier.  Please see "complications" below.
>>> 3.  Third, when the COMAD and pPOD sources have been successfully 
>>> integrated with the main Kepler system, I will delete the code under 
>>> src/org/nddp, this package having been superseded by the two sets of 
>>> checkins above.
>>> *** Complications  ***
>>> Ok, here's the conundrum.  The pPOD extension is mostly a set of 
>>> actors for phylogenetics analyses.  Prior to the Kepler 1.0 release 
>>> we provided a preview release 
>>> <http://daks.genomecenter.ucdavis.edu/kepler-ppod/> of the pPOD 
>>> actors (based on the COMAD framework) to the AToL 
>>> <http://atol.sdsc.edu/> community (see a poster 
>>> <http://daks.genomecenter.ucdavis.edu/kepler-ppod/poster.pdf> and 
>>> presentation 
>>> <http://daks.genomecenter.ucdavis.edu/kepler-ppod/Kepler-Ppod.pdf> 
>>> about the Kepler/ppod preview).  This was not meant for production 
>>> use, but rather as a way to get feedback from the community on our 
>>> work.  However, in the very near future we would like to release this 
>>> code as a package of actors and community-specific customizations to 
>>> the Kepler GUI.   Note that this release will not be targeted only at 
>>> an abstract community of folks who might possibly try it out, but 
>>> also at particular scientists who have asked for specific workflows 
>>> based on the pPOD package and who need them very badly for their own 
>>> research.)
>>> As you might expect, we do *not* want to:
>>> N1.  Wait for the next release of Kepler before distributing these 
>>> actors. N2.  Release a package of actors that has been tested only 
>>> against  the revision of Kepler that we happen to find at the trunk 
>>> of the Kepler repository on a particular day. Instead, we want to 
>>> (read 'must') release a package of actors that works with Kepler 
>>> 1.0.  When another release of Kepler is made (Kepler 1.1, say), we 
>>> will need to release a version of the pPOD actors that works with 
>>> *that* version of Kepler.  There are number of questions implied by 
>>> this need to release a package of actors designed to work with a 
>>> particular, supported release of Kepler:
>>> Q1. How do we build the pPOD actors against Kepler 1.0 rather than 
>>> against the current revision in the repository?
>>
>> One way to do this would be to have a build.xml file in your ppod 
>> source tree that just knows how to build ppod.  The build.xml file for 
>> kepler could be made to import the ppod build file when ppod needs to 
>> be built.  In the future, we could redesign the kepler build.xml file 
>> to allow importing different build files from the command line so that 
>> the build file itself does not need to be altered.
> 
> An extension-specific build file in each extension source tree sounds 
> like a good idea, as long as it doesn't require duplicating 
> functionality provided by the central build.xml file.  There is more to 
> do than build the Java sources, too.  Actor package extensions will have 
> their own set of kar source files that will be need to be jarred up the 
> same way kar sources for actors in the core of Kepler are, for example.  
> The build system ought to use the central support for building kar files 
> rather requiring that package builders reinvent or copy this 
> functionality.  (The hacked version of the build file we use at UCD 
> deals with this issue--more or less).
> 
>>
>>> Q2.  Where should we put the pPOD actors and support classes in the 
>>> Kepler repository directory structure?  If we put them in, say, 
>>> src/org/ppod (under kepler), then all of this code will be seen by 
>>> the build system and all kepler developers will find themselves 
>>> building code that is not necessarily meant to be compiled against 
>>> the current revision of Kepler in the repository.  If any classes the 
>>> pPOD package depends on elsewhere in Kepler are renamed, moved, 
>>> removed, or changed significantly, the pPOD code will no longer build 
>>> against the current revision of Kepler in the repository.  The build 
>>> will break.
>>
>> if the build file system is used, the ppod actors would not be built 
>> unless the user told the build system to build them.
> 
> So there would be a place in the repository for source code trees that 
> do not get compiled by default with the core of Kepler?  Excellent.
> 
>>
>>> A further complication is that once the 1.0-compatible version of the 
>>> pPOD package is out, we likely will want to begin building against 
>>> the latest revision of Kepler in the repository until the next 
>>> version of Kepler is out, at which point we'll want to build against 
>>> *that* supported version of Kepler.  In short, we need to be able to 
>>> choose which revision of Kepler in the repository we build the pPOD 
>>> package against, without disturbing folks working with the current 
>>> revision of Kepler.
>>
>> I think this could be done with ant.  you'll just need to have your 
>> build file make sure that the current checkout of kepler is the 
>> tag/branch that you want.
>>
> 
> Cool.
> 
>>> *** Solutions? ***
>>> S1.  On the surface this problem sounds like one that could be solved 
>>> with branches.  And I believe it possibly could, and that branches 
>>> likely will be involved in some way (for example, we will still want 
>>> to provide bug fixes to the 1.0-compatible version of the pPOD actors 
>>> when we're working on the 1.1-compatible release).  However, is this 
>>> how we want to treat the Kepler 1.0 release branch in the 
>>> repository?  Do we want to be checking hundreds of source files into 
>>> the Kepler 1.0 release branch indefinitely, and almost certainly 
>>> introducing new bugs and instabilities there?
>>
>> Kepler 1.0 is released, so we can no longer make changes to that 
>> branch and still call it 1.0.  This is not an option, imho.
> 
> I agree with this.
> 
>>
>>> S2.  An alternative would be to create a separate source tree for 
>>> pPOD in the svn repository (as a peer to kepler, for example).  The 
>>> build system would not include this 'extension' when compiling the 
>>> Kepler java sources.  Instead, we might build the pPOD package 
>>> against the Kepler 1.0 release jar itself.  This approach might scale 
>>> better and enable everyone providing their users 1.0-compatible 
>>> actors to share their work with the community through the Kepler 
>>> repository without requiring that their code work with the current 
>>> revision of Kepler.
>>
>> I kind of think of each directory in the repository as its own 
>> repository anyway.  I think the compilation/non-compilation of a given 
>> directory should be controlled by the build file(s).  I don't think 
>> its necessary to have an entirely new repository for extensions.  If 
>> this is something that is needed by the greater community, we could 
>> create a high level directory in the kepler repository for extensions 
>> (kepler/extension) and have a hierarchical build system built around 
>> that.
> 
> I agree a new repository is not needed.  A directory for hosting 
> extensions would be great.  I'd start checking in the COMAD and pPOD 
> extensions today if there were such a place in the repository!
> 
>>
>>> The problem with this second approach is that it might not be so easy 
>>> to build against the current revision of Kepler, should one choose to 
>>> do so (and as we likely will between releases of the pPOD package).  
>>> At least, not without building the Kepler jar each time.
>>
>> I think the build file in the extensions directory could easily build 
>> kepler to the version you need and put the jar file into the 
>> extensions dir for use.  then it wouldn't need to be built multiple 
>> times and you'd guarantee that you were building against the version 
>> of kepler you want.
> 
> Sounds good!
> 
>>
>>> S3.  A variant of S2 would be to enable developers to tell the build 
>>> system to optionally include extensions in the build.  With the 
>>> extensions stored as peers to the kepler directory in the repository, 
>>> developers could choose which extensions to check out, and specify 
>>> different branches or tags for Kepler and for each extension checked 
>>> out.
>>
>> I guess my approach above is similar to this except that it uses the 
>> main kepler repository instead of different repositories.
> 
> Yep, I don't see a need for different repositories for each extension.  
> On the other hand, it would be nice if the build system could build 
> extensions not stored in the central repository, too (e.g., extensions 
> stored in other repositories but that have been checked out out to a 
> developer's machine).
> 
> Tristan--do you have thoughts about this?
> 
>>
>> I'm kind of thinking off the cuff here so I hope my ideas don't seem 
>> too malformed.  Maybe we could have a call next week to chat about this.
>>
> 
> Let's definitely chat soon.  It'd be great to move quickly on this.
> 
> All--who wants to participate in that discussion?  Tristan?  Christopher 
> Tuot?
> 
> 
> Thanks, Chad!
> 
> Tim
> 
>