[kepler-dev] where do actor packages meant to work with Kepler 1.0 go in the repository?

Thu Jun 26 18:11:17 PDT 2008

Forgive me if i've misread something or make no sense, I'm currently at home
with the flu, and the old brain cogs aren't turning as smoothly as they
normally are.
I think having an extensions directory would work as a temporary solution,
but I don't think extension code should be checked into the trunk of kepler.
For me it would make more sense to have a ~/kepler-extensions/ directory
where you would commit your extension code to. i.e.
~/kepler-extensions/comad/trunk/ and ~/kepler-extensions/ppod/trunk/. From
there, you could check out the kepler code, then check out any extensions
you want into the extensions directory (which should have an svn:ignore
property set to it so no one can accidently commit their extension code to
the kepler tree). If each extension was based on a standardised build.xml
(i.e. they have targets to compile and clean etc with names that are the
same for each extension) then the main kepler build file could scan it's
extension directory to find any build.xml files and run the appropriate
targets on them.

Of course, each directory in subversion can be checked out independent of
it's parents, but I think taking advantage of this and putting extension
projects inside the kepler trunk would cause confusion. Every project should
be kept in it's own space, otherwise, how are you to tell what is part of
the kepler core project, and what is an extension project? Also, it's not
possible (maybe it is, but I imagine it's not easy) to cherry-pick out the
extensions you want when you do an svn checkout of the kepler trunk.
Checking extension code into the kepler trunk still forces extension
developers into the same release schedule as kepler, and on a new release
version of kepler, the kepler releasers have to ensure that all the
extensions are stable, otherwise not include them in the release.

I think something like what my first email proposed is a necessity, and
anything that gets decided here should only be a temporary solution until
the core infrastructure is sorted out. OSGI seems like a good candidate (I
think Christopher Tuot has some experience here). And an SDK of some form
still sounds like a good idea to me, but exactly what it would be
still alludes me (idea's are forming, but i haven't been able to grab onto
anything solid yet).

Cheers
-Tristan

On Fri, Jun 27, 2008 at 7:57 AM, Timothy McPhillips <tmcphillips at mac.com>
wrote:

> Hi Chad,
>
> This all sounds good.
>
> However, it occurs to me that one possible difficulty with having the
> extensions in the kepler repository itself is that it could be hard
> to mix and match local copies of extensions checked out of different
> repositories.  If a local copy of the source code for the core of
> kepler is in ~/kepler/trunk/src and the extensions are in ~/kepler/
> trunk/extensions, then including a copy of an extension stored
> elsewhere means you're checking out code from one repository into a
> directory that is version controlled by a different repository.  Is
> this a problem?
>
> Thanks!
>
> Tim
>
>
> On Jun 26, 2008, at 11:22 AM, Chad Berkley wrote:
>
> > I think we could make it general enough that you could build an
> > extension out of a directory within kepler or from a different
> > repository all together.  Like I said, I kind of view directories
> > in the repository as different repositories anyway.
> >
> > I definitely support making it so that functionality in the root
> > build.xml file is not redundant.  I think ant does this pretty
> > nicely. Our metacat product does something similar for the ecogrid
> > extensions.
> >
> > I'm available pretty much anytime next week monday - wednesday.
> > I'm busy most of Thursday and Friday is a holiday (Independence Day).
> >
> > chad
> >
> >
> > Timothy McPhillips wrote:
> >> Hi Chad,
> >> I like your suggestions.  I've added some questions and comments
> >> of my own:
> >> On Jun 26, 2008, at 9:18 AM, Chad Berkley wrote:
> >>> Hi Tim,
> >>>
> >>> See my thoughts below.
> >>>
> >>> Timothy McPhillips wrote:
> >>>> Dear All,
> >>>> With the Kepler 1.0 release out and the migration to svn
> >>>> complete I am about to begin migrating into the Kepler
> >>>> repository some code we've been developing here at UCD.  Some of
> >>>> these checkins will be relatively straightforward, but other
> >>>> code will present some integration, source code management, and
> >>>> build system challenges I'd really like to get everyone's input
> >>>> and help with.
> >>>> *** Upcoming checkins ***
> >>>> 1.  First, I will be committing a new version of the COMAD
> >>>> framework to src/org/kepler/comad.  The current version in
> >>>> Kepler is included under src/org/nddp.  The major difference
> >>>> with the new version is that it includes comprehensive support
> >>>> for recording provenance from COMAD workflows and storing the
> >>>> results as provenance-annotated 'trace' files.  2.  Second, we
> >>>> would like to begin committing the code specific to the pPOD
> >>>> extension to Kepler (phylogenetics actors and data types for the
> >>>> AToL community).  This integration will be trickier.  Please see
> >>>> "complications" below.
> >>>> 3.  Third, when the COMAD and pPOD sources have been
> >>>> successfully integrated with the main Kepler system, I will
> >>>> delete the code under src/org/nddp, this package having been
> >>>> superseded by the two sets of checkins above.
> >>>> *** Complications  ***
> >>>> Ok, here's the conundrum.  The pPOD extension is mostly a set of
> >>>> actors for phylogenetics analyses.  Prior to the Kepler 1.0
> >>>> release we provided a preview release <http://
> >>>> daks.genomecenter.ucdavis.edu/kepler-ppod/> of the pPOD actors
> >>>> (based on the COMAD framework) to the AToL <http://atol.sdsc.edu/
> >>>> > community (see a poster <http://daks.genomecenter.ucdavis.edu/
> >>>> kepler-ppod/poster.pdf> and presentation <http://
> >>>> daks.genomecenter.ucdavis.edu/kepler-ppod/Kepler-Ppod.pdf> about
> >>>> the Kepler/ppod preview).  This was not meant for production
> >>>> use, but rather as a way to get feedback from the community on
> >>>> our work.  However, in the very near future we would like to
> >>>> release this code as a package of actors and community-specific
> >>>> customizations to the Kepler GUI.   Note that this release will
> >>>> not be targeted only at an abstract community of folks who might
> >>>> possibly try it out, but also at particular scientists who have
> >>>> asked for specific workflows based on the pPOD package and who
> >>>> need them very badly for their own research.)
> >>>> As you might expect, we do *not* want to:
> >>>> N1.  Wait for the next release of Kepler before distributing
> >>>> these actors. N2.  Release a package of actors that has been
> >>>> tested only against  the revision of Kepler that we happen to
> >>>> find at the trunk of the Kepler repository on a particular day.
> >>>> Instead, we want to (read 'must') release a package of actors
> >>>> that works with Kepler 1.0.  When another release of Kepler is
> >>>> made (Kepler 1.1, say), we will need to release a version of the
> >>>> pPOD actors that works with *that* version of Kepler.  There are
> >>>> number of questions implied by this need to release a package of
> >>>> actors designed to work with a particular, supported release of
> >>>> Kepler:
> >>>> Q1. How do we build the pPOD actors against Kepler 1.0 rather
> >>>> than against the current revision in the repository?
> >>>
> >>> One way to do this would be to have a build.xml file in your ppod
> >>> source tree that just knows how to build ppod.  The build.xml
> >>> file for kepler could be made to import the ppod build file when
> >>> ppod needs to be built.  In the future, we could redesign the
> >>> kepler build.xml file to allow importing different build files
> >>> from the command line so that the build file itself does not need
> >>> to be altered.
> >> An extension-specific build file in each extension source tree
> >> sounds like a good idea, as long as it doesn't require duplicating
> >> functionality provided by the central build.xml file.  There is
> >> more to do than build the Java sources, too.  Actor package
> >> extensions will have their own set of kar source files that will
> >> be need to be jarred up the same way kar sources for actors in the
> >> core of Kepler are, for example.  The build system ought to use
> >> the central support for building kar files rather requiring that
> >> package builders reinvent or copy this functionality.  (The hacked
> >> version of the build file we use at UCD deals with this issue--
> >> more or less).
> >>>
> >>>> Q2.  Where should we put the pPOD actors and support classes in
> >>>> the Kepler repository directory structure?  If we put them in,
> >>>> say, src/org/ppod (under kepler), then all of this code will be
> >>>> seen by the build system and all kepler developers will find
> >>>> themselves building code that is not necessarily meant to be
> >>>> compiled against the current revision of Kepler in the
> >>>> repository.  If any classes the pPOD package depends on
> >>>> elsewhere in Kepler are renamed, moved, removed, or changed
> >>>> significantly, the pPOD code will no longer build against the
> >>>> current revision of Kepler in the repository.  The build will
> >>>> break.
> >>>
> >>> if the build file system is used, the ppod actors would not be
> >>> built unless the user told the build system to build them.
> >> So there would be a place in the repository for source code trees
> >> that do not get compiled by default with the core of Kepler?
> >> Excellent.
> >>>
> >>>> A further complication is that once the 1.0-compatible version
> >>>> of the pPOD package is out, we likely will want to begin
> >>>> building against the latest revision of Kepler in the repository
> >>>> until the next version of Kepler is out, at which point we'll
> >>>> want to build against *that* supported version of Kepler.  In
> >>>> short, we need to be able to choose which revision of Kepler in
> >>>> the repository we build the pPOD package against, without
> >>>> disturbing folks working with the current revision of Kepler.
> >>>
> >>> I think this could be done with ant.  you'll just need to have
> >>> your build file make sure that the current checkout of kepler is
> >>> the tag/branch that you want.
> >>>
> >> Cool.
> >>>> *** Solutions? ***
> >>>> S1.  On the surface this problem sounds like one that could be
> >>>> solved with branches.  And I believe it possibly could, and that
> >>>> branches likely will be involved in some way (for example, we
> >>>> will still want to provide bug fixes to the 1.0-compatible
> >>>> version of the pPOD actors when we're working on the 1.1-
> >>>> compatible release).  However, is this how we want to treat the
> >>>> Kepler 1.0 release branch in the repository?  Do we want to be
> >>>> checking hundreds of source files into the Kepler 1.0 release
> >>>> branch indefinitely, and almost certainly introducing new bugs
> >>>> and instabilities there?
> >>>
> >>> Kepler 1.0 is released, so we can no longer make changes to that
> >>> branch and still call it 1.0.  This is not an option, imho.
> >> I agree with this.
> >>>
> >>>> S2.  An alternative would be to create a separate source tree
> >>>> for pPOD in the svn repository (as a peer to kepler, for
> >>>> example).  The build system would not include this 'extension'
> >>>> when compiling the Kepler java sources.  Instead, we might build
> >>>> the pPOD package against the Kepler 1.0 release jar itself.
> >>>> This approach might scale better and enable everyone providing
> >>>> their users 1.0-compatible actors to share their work with the
> >>>> community through the Kepler repository without requiring that
> >>>> their code work with the current revision of Kepler.
> >>>
> >>> I kind of think of each directory in the repository as its own
> >>> repository anyway.  I think the compilation/non-compilation of a
> >>> given directory should be controlled by the build file(s).  I
> >>> don't think its necessary to have an entirely new repository for
> >>> extensions.  If this is something that is needed by the greater
> >>> community, we could create a high level directory in the kepler
> >>> repository for extensions (kepler/extension) and have a
> >>> hierarchical build system built around that.
> >> I agree a new repository is not needed.  A directory for hosting
> >> extensions would be great.  I'd start checking in the COMAD and
> >> pPOD extensions today if there were such a place in the repository!
> >>>
> >>>> The problem with this second approach is that it might not be so
> >>>> easy to build against the current revision of Kepler, should one
> >>>> choose to do so (and as we likely will between releases of the
> >>>> pPOD package).  At least, not without building the Kepler jar
> >>>> each time.
> >>>
> >>> I think the build file in the extensions directory could easily
> >>> build kepler to the version you need and put the jar file into
> >>> the extensions dir for use.  then it wouldn't need to be built
> >>> multiple times and you'd guarantee that you were building against
> >>> the version of kepler you want.
> >> Sounds good!
> >>>
> >>>> S3.  A variant of S2 would be to enable developers to tell the
> >>>> build system to optionally include extensions in the build.
> >>>> With the extensions stored as peers to the kepler directory in
> >>>> the repository, developers could choose which extensions to
> >>>> check out, and specify different branches or tags for Kepler and
> >>>> for each extension checked out.
> >>>
> >>> I guess my approach above is similar to this except that it uses
> >>> the main kepler repository instead of different repositories.
> >> Yep, I don't see a need for different repositories for each
> >> extension.  On the other hand, it would be nice if the build
> >> system could build extensions not stored in the central
> >> repository, too (e.g., extensions stored in other repositories but
> >> that have been checked out out to a developer's machine).
> >> Tristan--do you have thoughts about this?
> >>>
> >>> I'm kind of thinking off the cuff here so I hope my ideas don't
> >>> seem too malformed.  Maybe we could have a call next week to chat
> >>> about this.
> >>>
> >> Let's definitely chat soon.  It'd be great to move quickly on this.
> >> All--who wants to participate in that discussion?  Tristan?
> >> Christopher Tuot?
> >> Thanks, Chad!
> >> Tim
>
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at ecoinformatics.org
> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/kepler-dev
>

-- 
Tristan King
Research Officer,
eResearch Centre
James Cook University, Townsville Qld 4811
Australia

Phone: +61747816902
E-mail: tristan.king at jcu.edu.au www: http://eresearch.jcu.edu.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mercury.nceas.ucsb.edu/ecoinformatics/pipermail/kepler-dev/attachments/20080627/059cc4b6/attachment-0001.htm 