[kepler-dev] where do actor packages meant to work with Kepler 1.0 go in the repository?

[Chad Berkley]

It seems like if you're mixing and matching like that, then the atomic 
extensions should provide a jar file for other extensions to use.  That 
having been said, it's easy to include various directories with ant.  I 
don't really see it as a problem.

chad


Timothy McPhillips wrote:
> Hi Chad,
> 
> This all sounds good.
> 
> However, it occurs to me that one possible difficulty with having the 
> extensions in the kepler repository itself is that it could be hard to 
> mix and match local copies of extensions checked out of different 
> repositories.  If a local copy of the source code for the core of kepler 
> is in ~/kepler/trunk/src and the extensions are in 
> ~/kepler/trunk/extensions, then including a copy of an extension stored 
> elsewhere means you're checking out code from one repository into a 
> directory that is version controlled by a different repository.  Is this 
> a problem?
> 
> Thanks!
> 
> Tim
> 
> 
> On Jun 26, 2008, at 11:22 AM, Chad Berkley wrote:
> 
>> I think we could make it general enough that you could build an 
>> extension out of a directory within kepler or from a different 
>> repository all together.  Like I said, I kind of view directories in 
>> the repository as different repositories anyway.
>>
>> I definitely support making it so that functionality in the root 
>> build.xml file is not redundant.  I think ant does this pretty nicely. 
>> Our metacat product does something similar for the ecogrid extensions.
>>
>> I'm available pretty much anytime next week monday - wednesday.  I'm 
>> busy most of Thursday and Friday is a holiday (Independence Day).
>>
>> chad
>>
>>
>> Timothy McPhillips wrote:
>>> Hi Chad,
>>> I like your suggestions.  I've added some questions and comments of 
>>> my own:
>>> On Jun 26, 2008, at 9:18 AM, Chad Berkley wrote:
>>>> Hi Tim,
>>>>
>>>> See my thoughts below.
>>>>
>>>> Timothy McPhillips wrote:
>>>>> Dear All,
>>>>> With the Kepler 1.0 release out and the migration to svn complete I 
>>>>> am about to begin migrating into the Kepler repository some code 
>>>>> we've been developing here at UCD.  Some of these checkins will be 
>>>>> relatively straightforward, but other code will present some 
>>>>> integration, source code management, and build system challenges 
>>>>> I'd really like to get everyone's input and help with.
>>>>> *** Upcoming checkins ***
>>>>> 1.  First, I will be committing a new version of the COMAD 
>>>>> framework to src/org/kepler/comad.  The current version in Kepler 
>>>>> is included under src/org/nddp.  The major difference with the new 
>>>>> version is that it includes comprehensive support for recording 
>>>>> provenance from COMAD workflows and storing the results as 
>>>>> provenance-annotated 'trace' files.  2.  Second, we would like to 
>>>>> begin committing the code specific to the pPOD extension to Kepler 
>>>>> (phylogenetics actors and data types for the AToL community).  This 
>>>>> integration will be trickier.  Please see "complications" below.
>>>>> 3.  Third, when the COMAD and pPOD sources have been successfully 
>>>>> integrated with the main Kepler system, I will delete the code 
>>>>> under src/org/nddp, this package having been superseded by the two 
>>>>> sets of checkins above.
>>>>> *** Complications  ***
>>>>> Ok, here's the conundrum.  The pPOD extension is mostly a set of 
>>>>> actors for phylogenetics analyses.  Prior to the Kepler 1.0 release 
>>>>> we provided a preview release 
>>>>> <http://daks.genomecenter.ucdavis.edu/kepler-ppod/> of the pPOD 
>>>>> actors (based on the COMAD framework) to the AToL 
>>>>> <http://atol.sdsc.edu/> community (see a poster 
>>>>> <http://daks.genomecenter.ucdavis.edu/kepler-ppod/poster.pdf> and 
>>>>> presentation 
>>>>> <http://daks.genomecenter.ucdavis.edu/kepler-ppod/Kepler-Ppod.pdf> 
>>>>> about the Kepler/ppod preview).  This was not meant for production 
>>>>> use, but rather as a way to get feedback from the community on our 
>>>>> work.  However, in the very near future we would like to release 
>>>>> this code as a package of actors and community-specific 
>>>>> customizations to the Kepler GUI.   Note that this release will not 
>>>>> be targeted only at an abstract community of folks who might 
>>>>> possibly try it out, but also at particular scientists who have 
>>>>> asked for specific workflows based on the pPOD package and who need 
>>>>> them very badly for their own research.)
>>>>> As you might expect, we do *not* want to:
>>>>> N1.  Wait for the next release of Kepler before distributing these 
>>>>> actors. N2.  Release a package of actors that has been tested only 
>>>>> against  the revision of Kepler that we happen to find at the trunk 
>>>>> of the Kepler repository on a particular day. Instead, we want to 
>>>>> (read 'must') release a package of actors that works with Kepler 
>>>>> 1.0.  When another release of Kepler is made (Kepler 1.1, say), we 
>>>>> will need to release a version of the pPOD actors that works with 
>>>>> *that* version of Kepler.  There are number of questions implied by 
>>>>> this need to release a package of actors designed to work with a 
>>>>> particular, supported release of Kepler:
>>>>> Q1. How do we build the pPOD actors against Kepler 1.0 rather than 
>>>>> against the current revision in the repository?
>>>>
>>>> One way to do this would be to have a build.xml file in your ppod 
>>>> source tree that just knows how to build ppod.  The build.xml file 
>>>> for kepler could be made to import the ppod build file when ppod 
>>>> needs to be built.  In the future, we could redesign the kepler 
>>>> build.xml file to allow importing different build files from the 
>>>> command line so that the build file itself does not need to be altered.
>>> An extension-specific build file in each extension source tree sounds 
>>> like a good idea, as long as it doesn't require duplicating 
>>> functionality provided by the central build.xml file.  There is more 
>>> to do than build the Java sources, too.  Actor package extensions 
>>> will have their own set of kar source files that will be need to be 
>>> jarred up the same way kar sources for actors in the core of Kepler 
>>> are, for example.  The build system ought to use the central support 
>>> for building kar files rather requiring that package builders 
>>> reinvent or copy this functionality.  (The hacked version of the 
>>> build file we use at UCD deals with this issue--more or less).
>>>>
>>>>> Q2.  Where should we put the pPOD actors and support classes in the 
>>>>> Kepler repository directory structure?  If we put them in, say, 
>>>>> src/org/ppod (under kepler), then all of this code will be seen by 
>>>>> the build system and all kepler developers will find themselves 
>>>>> building code that is not necessarily meant to be compiled against 
>>>>> the current revision of Kepler in the repository.  If any classes 
>>>>> the pPOD package depends on elsewhere in Kepler are renamed, moved, 
>>>>> removed, or changed significantly, the pPOD code will no longer 
>>>>> build against the current revision of Kepler in the repository.  
>>>>> The build will break.
>>>>
>>>> if the build file system is used, the ppod actors would not be built 
>>>> unless the user told the build system to build them.
>>> So there would be a place in the repository for source code trees 
>>> that do not get compiled by default with the core of Kepler?  Excellent.
>>>>
>>>>> A further complication is that once the 1.0-compatible version of 
>>>>> the pPOD package is out, we likely will want to begin building 
>>>>> against the latest revision of Kepler in the repository until the 
>>>>> next version of Kepler is out, at which point we'll want to build 
>>>>> against *that* supported version of Kepler.  In short, we need to 
>>>>> be able to choose which revision of Kepler in the repository we 
>>>>> build the pPOD package against, without disturbing folks working 
>>>>> with the current revision of Kepler.
>>>>
>>>> I think this could be done with ant.  you'll just need to have your 
>>>> build file make sure that the current checkout of kepler is the 
>>>> tag/branch that you want.
>>>>
>>> Cool.
>>>>> *** Solutions? ***
>>>>> S1.  On the surface this problem sounds like one that could be 
>>>>> solved with branches.  And I believe it possibly could, and that 
>>>>> branches likely will be involved in some way (for example, we will 
>>>>> still want to provide bug fixes to the 1.0-compatible version of 
>>>>> the pPOD actors when we're working on the 1.1-compatible release).  
>>>>> However, is this how we want to treat the Kepler 1.0 release branch 
>>>>> in the repository?  Do we want to be checking hundreds of source 
>>>>> files into the Kepler 1.0 release branch indefinitely, and almost 
>>>>> certainly introducing new bugs and instabilities there?
>>>>
>>>> Kepler 1.0 is released, so we can no longer make changes to that 
>>>> branch and still call it 1.0.  This is not an option, imho.
>>> I agree with this.
>>>>
>>>>> S2.  An alternative would be to create a separate source tree for 
>>>>> pPOD in the svn repository (as a peer to kepler, for example).  The 
>>>>> build system would not include this 'extension' when compiling the 
>>>>> Kepler java sources.  Instead, we might build the pPOD package 
>>>>> against the Kepler 1.0 release jar itself.  This approach might 
>>>>> scale better and enable everyone providing their users 
>>>>> 1.0-compatible actors to share their work with the community 
>>>>> through the Kepler repository without requiring that their code 
>>>>> work with the current revision of Kepler.
>>>>
>>>> I kind of think of each directory in the repository as its own 
>>>> repository anyway.  I think the compilation/non-compilation of a 
>>>> given directory should be controlled by the build file(s).  I don't 
>>>> think its necessary to have an entirely new repository for 
>>>> extensions.  If this is something that is needed by the greater 
>>>> community, we could create a high level directory in the kepler 
>>>> repository for extensions (kepler/extension) and have a hierarchical 
>>>> build system built around that.
>>> I agree a new repository is not needed.  A directory for hosting 
>>> extensions would be great.  I'd start checking in the COMAD and pPOD 
>>> extensions today if there were such a place in the repository!
>>>>
>>>>> The problem with this second approach is that it might not be so 
>>>>> easy to build against the current revision of Kepler, should one 
>>>>> choose to do so (and as we likely will between releases of the pPOD 
>>>>> package).  At least, not without building the Kepler jar each time.
>>>>
>>>> I think the build file in the extensions directory could easily 
>>>> build kepler to the version you need and put the jar file into the 
>>>> extensions dir for use.  then it wouldn't need to be built multiple 
>>>> times and you'd guarantee that you were building against the version 
>>>> of kepler you want.
>>> Sounds good!
>>>>
>>>>> S3.  A variant of S2 would be to enable developers to tell the 
>>>>> build system to optionally include extensions in the build.  With 
>>>>> the extensions stored as peers to the kepler directory in the 
>>>>> repository, developers could choose which extensions to check out, 
>>>>> and specify different branches or tags for Kepler and for each 
>>>>> extension checked out.
>>>>
>>>> I guess my approach above is similar to this except that it uses the 
>>>> main kepler repository instead of different repositories.
>>> Yep, I don't see a need for different repositories for each 
>>> extension.  On the other hand, it would be nice if the build system 
>>> could build extensions not stored in the central repository, too 
>>> (e.g., extensions stored in other repositories but that have been 
>>> checked out out to a developer's machine).
>>> Tristan--do you have thoughts about this?
>>>>
>>>> I'm kind of thinking off the cuff here so I hope my ideas don't seem 
>>>> too malformed.  Maybe we could have a call next week to chat about 
>>>> this.
>>>>
>>> Let's definitely chat soon.  It'd be great to move quickly on this.
>>> All--who wants to participate in that discussion?  Tristan?  
>>> Christopher Tuot?
>>> Thanks, Chad!
>>> Tim
>