[kepler-dev] where do actor packages meant to work with Kepler 1.0 go in the repository?

[Timothy McPhillips]

Hi Chad,

Cool, thanks!  I'm not sure I follow you completely, but it sounds  
like you're saying that developing multiple extensions might be best  
handled by holding all but one constant (each of the fixed extensions  
held in their own jars) at a time.  Or that we could just specify a  
list of extension directories to ant, say.

I'm not terribly worried about how we solve this problem (although it  
sounds like Tristan has a different proposal).  I'd just like for us  
all to select a workable solution soon.

What would you propose for an agenda for a telecon next week?   
Perhaps everyone interested in these issues could participate and:

(1) Those with specific solutions in mind (repository structures and  
assumptions, build system modfications, etc) could propose them in  
detail.
(2) Those of us who will use the system to develop Kepler extensions  
(and provide them to users) could critique those solutions on the  
basis of whether they meet our needs and discard those that don't.
(3)  We could all vote on the solutions that make it past step 2.
(4)  We could then decide how and when to implement this solution.

On the other hand, all of this possibly could be done on kepler-dev.  
(That would be my preference.)

Thoughts?

Tim

P.S. One situation in which the mix-and-match problem (working with  
extensions from more than one repository) comes up is when developing  
actors and workflows for scientists who don't want their research  
ideas and processes exposed to the world before they're ready to  
report and share them--a common enough scenario.


On Jun 27, 2008, at 11:44 AM, Chad Berkley wrote:

> It seems like if you're mixing and matching like that, then the  
> atomic extensions should provide a jar file for other extensions to  
> use.  That having been said, it's easy to include various  
> directories with ant.  I don't really see it as a problem.
>
> chad
>
>
> Timothy McPhillips wrote:
>> Hi Chad,
>> This all sounds good.
>> However, it occurs to me that one possible difficulty with having  
>> the extensions in the kepler repository itself is that it could be  
>> hard to mix and match local copies of extensions checked out of  
>> different repositories.  If a local copy of the source code for  
>> the core of kepler is in ~/kepler/trunk/src and the extensions are  
>> in ~/kepler/trunk/extensions, then including a copy of an  
>> extension stored elsewhere means you're checking out code from one  
>> repository into a directory that is version controlled by a  
>> different repository.  Is this a problem?
>> Thanks!
>> Tim
>> On Jun 26, 2008, at 11:22 AM, Chad Berkley wrote:
>>> I think we could make it general enough that you could build an  
>>> extension out of a directory within kepler or from a different  
>>> repository all together.  Like I said, I kind of view directories  
>>> in the repository as different repositories anyway.
>>>
>>> I definitely support making it so that functionality in the root  
>>> build.xml file is not redundant.  I think ant does this pretty  
>>> nicely. Our metacat product does something similar for the  
>>> ecogrid extensions.
>>>
>>> I'm available pretty much anytime next week monday - wednesday.   
>>> I'm busy most of Thursday and Friday is a holiday (Independence  
>>> Day).
>>>
>>> chad
>>>
>>>
>>> Timothy McPhillips wrote:
>>>> Hi Chad,
>>>> I like your suggestions.  I've added some questions and comments  
>>>> of my own:
>>>> On Jun 26, 2008, at 9:18 AM, Chad Berkley wrote:
>>>>> Hi Tim,
>>>>>
>>>>> See my thoughts below.
>>>>>
>>>>> Timothy McPhillips wrote:
>>>>>> Dear All,
>>>>>> With the Kepler 1.0 release out and the migration to svn  
>>>>>> complete I am about to begin migrating into the Kepler  
>>>>>> repository some code we've been developing here at UCD.  Some  
>>>>>> of these checkins will be relatively straightforward, but  
>>>>>> other code will present some integration, source code  
>>>>>> management, and build system challenges I'd really like to get  
>>>>>> everyone's input and help with.
>>>>>> *** Upcoming checkins ***
>>>>>> 1.  First, I will be committing a new version of the COMAD  
>>>>>> framework to src/org/kepler/comad.  The current version in  
>>>>>> Kepler is included under src/org/nddp.  The major difference  
>>>>>> with the new version is that it includes comprehensive support  
>>>>>> for recording provenance from COMAD workflows and storing the  
>>>>>> results as provenance-annotated 'trace' files.  2.  Second, we  
>>>>>> would like to begin committing the code specific to the pPOD  
>>>>>> extension to Kepler (phylogenetics actors and data types for  
>>>>>> the AToL community).  This integration will be trickier.   
>>>>>> Please see "complications" below.
>>>>>> 3.  Third, when the COMAD and pPOD sources have been  
>>>>>> successfully integrated with the main Kepler system, I will  
>>>>>> delete the code under src/org/nddp, this package having been  
>>>>>> superseded by the two sets of checkins above.
>>>>>> *** Complications  ***
>>>>>> Ok, here's the conundrum.  The pPOD extension is mostly a set  
>>>>>> of actors for phylogenetics analyses.  Prior to the Kepler 1.0  
>>>>>> release we provided a preview release <http:// 
>>>>>> daks.genomecenter.ucdavis.edu/kepler-ppod/> of the pPOD actors  
>>>>>> (based on the COMAD framework) to the AToL <http:// 
>>>>>> atol.sdsc.edu/> community (see a poster <http:// 
>>>>>> daks.genomecenter.ucdavis.edu/kepler-ppod/poster.pdf> and  
>>>>>> presentation <http://daks.genomecenter.ucdavis.edu/kepler-ppod/ 
>>>>>> Kepler-Ppod.pdf> about the Kepler/ppod preview).  This was not  
>>>>>> meant for production use, but rather as a way to get feedback  
>>>>>> from the community on our work.  However, in the very near  
>>>>>> future we would like to release this code as a package of  
>>>>>> actors and community-specific customizations to the Kepler  
>>>>>> GUI.   Note that this release will not be targeted only at an  
>>>>>> abstract community of folks who might possibly try it out, but  
>>>>>> also at particular scientists who have asked for specific  
>>>>>> workflows based on the pPOD package and who need them very  
>>>>>> badly for their own research.)
>>>>>> As you might expect, we do *not* want to:
>>>>>> N1.  Wait for the next release of Kepler before distributing  
>>>>>> these actors. N2.  Release a package of actors that has been  
>>>>>> tested only against  the revision of Kepler that we happen to  
>>>>>> find at the trunk of the Kepler repository on a particular  
>>>>>> day. Instead, we want to (read 'must') release a package of  
>>>>>> actors that works with Kepler 1.0.  When another release of  
>>>>>> Kepler is made (Kepler 1.1, say), we will need to release a  
>>>>>> version of the pPOD actors that works with *that* version of  
>>>>>> Kepler.  There are number of questions implied by this need to  
>>>>>> release a package of actors designed to work with a  
>>>>>> particular, supported release of Kepler:
>>>>>> Q1. How do we build the pPOD actors against Kepler 1.0 rather  
>>>>>> than against the current revision in the repository?
>>>>>
>>>>> One way to do this would be to have a build.xml file in your  
>>>>> ppod source tree that just knows how to build ppod.  The  
>>>>> build.xml file for kepler could be made to import the ppod  
>>>>> build file when ppod needs to be built.  In the future, we  
>>>>> could redesign the kepler build.xml file to allow importing  
>>>>> different build files from the command line so that the build  
>>>>> file itself does not need to be altered.
>>>> An extension-specific build file in each extension source tree  
>>>> sounds like a good idea, as long as it doesn't require  
>>>> duplicating functionality provided by the central build.xml  
>>>> file.  There is more to do than build the Java sources, too.   
>>>> Actor package extensions will have their own set of kar source  
>>>> files that will be need to be jarred up the same way kar sources  
>>>> for actors in the core of Kepler are, for example.  The build  
>>>> system ought to use the central support for building kar files  
>>>> rather requiring that package builders reinvent or copy this  
>>>> functionality.  (The hacked version of the build file we use at  
>>>> UCD deals with this issue--more or less).
>>>>>
>>>>>> Q2.  Where should we put the pPOD actors and support classes  
>>>>>> in the Kepler repository directory structure?  If we put them  
>>>>>> in, say, src/org/ppod (under kepler), then all of this code  
>>>>>> will be seen by the build system and all kepler developers  
>>>>>> will find themselves building code that is not necessarily  
>>>>>> meant to be compiled against the current revision of Kepler in  
>>>>>> the repository.  If any classes the pPOD package depends on  
>>>>>> elsewhere in Kepler are renamed, moved, removed, or changed  
>>>>>> significantly, the pPOD code will no longer build against the  
>>>>>> current revision of Kepler in the repository.  The build will  
>>>>>> break.
>>>>>
>>>>> if the build file system is used, the ppod actors would not be  
>>>>> built unless the user told the build system to build them.
>>>> So there would be a place in the repository for source code  
>>>> trees that do not get compiled by default with the core of  
>>>> Kepler?  Excellent.
>>>>>
>>>>>> A further complication is that once the 1.0-compatible version  
>>>>>> of the pPOD package is out, we likely will want to begin  
>>>>>> building against the latest revision of Kepler in the  
>>>>>> repository until the next version of Kepler is out, at which  
>>>>>> point we'll want to build against *that* supported version of  
>>>>>> Kepler.  In short, we need to be able to choose which revision  
>>>>>> of Kepler in the repository we build the pPOD package against,  
>>>>>> without disturbing folks working with the current revision of  
>>>>>> Kepler.
>>>>>
>>>>> I think this could be done with ant.  you'll just need to have  
>>>>> your build file make sure that the current checkout of kepler  
>>>>> is the tag/branch that you want.
>>>>>
>>>> Cool.
>>>>>> *** Solutions? ***
>>>>>> S1.  On the surface this problem sounds like one that could be  
>>>>>> solved with branches.  And I believe it possibly could, and  
>>>>>> that branches likely will be involved in some way (for  
>>>>>> example, we will still want to provide bug fixes to the 1.0- 
>>>>>> compatible version of the pPOD actors when we're working on  
>>>>>> the 1.1-compatible release).  However, is this how we want to  
>>>>>> treat the Kepler 1.0 release branch in the repository?  Do we  
>>>>>> want to be checking hundreds of source files into the Kepler  
>>>>>> 1.0 release branch indefinitely, and almost certainly  
>>>>>> introducing new bugs and instabilities there?
>>>>>
>>>>> Kepler 1.0 is released, so we can no longer make changes to  
>>>>> that branch and still call it 1.0.  This is not an option, imho.
>>>> I agree with this.
>>>>>
>>>>>> S2.  An alternative would be to create a separate source tree  
>>>>>> for pPOD in the svn repository (as a peer to kepler, for  
>>>>>> example).  The build system would not include this 'extension'  
>>>>>> when compiling the Kepler java sources.  Instead, we might  
>>>>>> build the pPOD package against the Kepler 1.0 release jar  
>>>>>> itself.  This approach might scale better and enable everyone  
>>>>>> providing their users 1.0-compatible actors to share their  
>>>>>> work with the community through the Kepler repository without  
>>>>>> requiring that their code work with the current revision of  
>>>>>> Kepler.
>>>>>
>>>>> I kind of think of each directory in the repository as its own  
>>>>> repository anyway.  I think the compilation/non-compilation of  
>>>>> a given directory should be controlled by the build file(s).  I  
>>>>> don't think its necessary to have an entirely new repository  
>>>>> for extensions.  If this is something that is needed by the  
>>>>> greater community, we could create a high level directory in  
>>>>> the kepler repository for extensions (kepler/extension) and  
>>>>> have a hierarchical build system built around that.
>>>> I agree a new repository is not needed.  A directory for hosting  
>>>> extensions would be great.  I'd start checking in the COMAD and  
>>>> pPOD extensions today if there were such a place in the repository!
>>>>>
>>>>>> The problem with this second approach is that it might not be  
>>>>>> so easy to build against the current revision of Kepler,  
>>>>>> should one choose to do so (and as we likely will between  
>>>>>> releases of the pPOD package).  At least, not without building  
>>>>>> the Kepler jar each time.
>>>>>
>>>>> I think the build file in the extensions directory could easily  
>>>>> build kepler to the version you need and put the jar file into  
>>>>> the extensions dir for use.  then it wouldn't need to be built  
>>>>> multiple times and you'd guarantee that you were building  
>>>>> against the version of kepler you want.
>>>> Sounds good!
>>>>>
>>>>>> S3.  A variant of S2 would be to enable developers to tell the  
>>>>>> build system to optionally include extensions in the build.   
>>>>>> With the extensions stored as peers to the kepler directory in  
>>>>>> the repository, developers could choose which extensions to  
>>>>>> check out, and specify different branches or tags for Kepler  
>>>>>> and for each extension checked out.
>>>>>
>>>>> I guess my approach above is similar to this except that it  
>>>>> uses the main kepler repository instead of different repositories.
>>>> Yep, I don't see a need for different repositories for each  
>>>> extension.  On the other hand, it would be nice if the build  
>>>> system could build extensions not stored in the central  
>>>> repository, too (e.g., extensions stored in other repositories  
>>>> but that have been checked out out to a developer's machine).
>>>> Tristan--do you have thoughts about this?
>>>>>
>>>>> I'm kind of thinking off the cuff here so I hope my ideas don't  
>>>>> seem too malformed.  Maybe we could have a call next week to  
>>>>> chat about this.
>>>>>
>>>> Let's definitely chat soon.  It'd be great to move quickly on this.
>>>> All--who wants to participate in that discussion?  Tristan?   
>>>> Christopher Tuot?
>>>> Thanks, Chad!
>>>> Tim