[kepler-dev] where do actor packages meant to work with Kepler 1.0 go in the repository?
Timothy McPhillips
tmcphillips at mac.com
Thu Jun 26 10:47:28 PDT 2008
Hi Tristan,
Thanks for your thoughts. They do clarify things!
I realize now that you are correct--the COMAD stuff should be thought
of as an extension, too, for the simple reason that we will continue
to develop it between releases of Kepler, and will need the pPOD and
other actor packages that depend on COMAD to be able to use revisions
more current than the one in the latest Kepler release. At the same
time, we do not necessarily want pPOD and COMAD development to
proceed in lockstep either. So they need to be two different
extensions.
I like the idea of using the actor repository to enable development
teams to distribute their actor packages independently of the Kepler
release schedule and use metadata to indicate what packages work with
what versions of Kepler. The key requirements here seem to be:
R1. Make it easy for developers to upload to the repository packages
of actors meant to work together, not just individual actors.
R2. Enable upload of library extension packages that do not
necessarily contain any actors at all, but that other packages might
depend on.
R3. Enable (require?) package contributors to specify what version
(s) of Kepler each package works with.
R4. Enable package contributors to specify what other packages
(library extensions like COMAD included) a package requires.
On the user end, the requirements might be that one be able to
R5. Find and download a package of actors that works with the
version of Kepler one is using.
R6. Download all Kepler library extensions that an actor package
depends on.
R7. Configure Kepler to use particular packages of actors and the
library extensions they require.
Tristan, have you thought more about what the functions of a Kepler
SDK would be, and how one would use it? I definitely like the idea
of enabling people to develop with an SDK that treats the current
release of Kepler as an (unchanging) black box and doesn't require
that all of Kepler be built each time you compile your own stuff.
And I agree all of the above can be treated independently of where
the source code for actor packages and extensions are stored, as long
as the the storage approach effectively decouples development and
release of these packages from the core of Kepler and from each
other. At the moment, the standard build framework requires that
code developed for use with Kepler be stored in the Kepler source
code tree, and that is very limiting.
Cheers,
Tim
On Jun 25, 2008, at 9:04 PM, Tristan King wrote:
> I think this is testament to the requirement that actors (i.e.
> pPOD) and library extentions i.e. (COMAD) not be part of the core
> kepler build and they should be stored in their own project, and
> built with reference to the kepler/ptolemy jars rather than built
> into them. However, there is obviously a lot of work to be done
> before that's possible (I spent half an hour trying to separate the
> ptolemy.actor.lib package out from ptolemy, but there are too many
> other packages that require it to compile so i gave up rather
> quickly).
>
> What I think needs to happen after that, is the actor repository
> needs to be revived and revamped. Kepler should be shipped with
> only a core set of actors, but have an option from kepler to browse
> all the libraries available in the repository and download the ones
> that want. metadata can be included to say if the packages are
> untested unstable or stable for the specific version of kepler the
> user is using, whether the package is certified by kepler or not
> and anything else that could be useful. Developers of actor
> packages can then simply upload their packages to the repository
> and not worry about getting it into a specific release of kepler.
>
> This doesn't really solve where to stick the code tho. But I think
> once my proposed infrastructure was put into place, then it isn't a
> big deal where the code is stored, as the repository could link to
> the packages home page where a developer could find the source code.
>
> I could throw out a few possible solutions, but i can't think of
> any that I think of any that i would actually suggest doing. In my
> view, something like what i've suggested above is the only good
> solution.
>
> Kepler SDK anyone? (the thought just popped into my head i'll
> ponder on that one some more)
>
> Sorry I can't be of any real help :)
> -Tristan
>
> On Thu, Jun 26, 2008 at 12:59 PM, Timothy McPhillips
> <tmcphillips at mac.com> wrote:
> Dear All,
>
> With the Kepler 1.0 release out and the migration to svn complete I
> am about to begin migrating into the Kepler repository some code
> we've been developing here at UCD. Some of these checkins will be
> relatively straightforward, but other code will present some
> integration, source code management, and build system challenges
> I'd really like to get everyone's input and help with.
>
>
> *** Upcoming checkins ***
>
> 1. First, I will be committing a new version of the COMAD
> framework to src/org/kepler/comad. The current version in Kepler
> is included under src/org/nddp. The major difference with the new
> version is that it includes comprehensive support for recording
> provenance from COMAD workflows and storing the results as
> provenance-annotated 'trace' files.
>
> 2. Second, we would like to begin committing the code specific to
> the pPOD extension to Kepler (phylogenetics actors and data types
> for the AToL community). This integration will be trickier.
> Please see "complications" below.
>
> 3. Third, when the COMAD and pPOD sources have been successfully
> integrated with the main Kepler system, I will delete the code
> under src/org/nddp, this package having been superseded by the two
> sets of checkins above.
>
>
> *** Complications ***
>
> Ok, here's the conundrum. The pPOD extension is mostly a set of
> actors for phylogenetics analyses. Prior to the Kepler 1.0 release
> we provided a preview release of the pPOD actors (based on the
> COMAD framework) to the AToL community (see a poster and
> presentation about the Kepler/ppod preview). This was not meant
> for production use, but rather as a way to get feedback from the
> community on our work. However, in the very near future we would
> like to release this code as a package of actors and community-
> specific customizations to the Kepler GUI. Note that this release
> will not be targeted only at an abstract community of folks who
> might possibly try it out, but also at particular scientists who
> have asked for specific workflows based on the pPOD package and who
> need them very badly for their own research.)
>
> As you might expect, we do *not* want to:
>
> N1. Wait for the next release of Kepler before distributing these
> actors.
>
> N2. Release a package of actors that has been tested only against
> the revision of Kepler that we happen to find at the trunk of the
> Kepler repository on a particular day.
>
> Instead, we want to (read 'must') release a package of actors that
> works with Kepler 1.0. When another release of Kepler is made
> (Kepler 1.1, say), we will need to release a version of the pPOD
> actors that works with *that* version of Kepler.
>
> There are number of questions implied by this need to release a
> package of actors designed to work with a particular, supported
> release of Kepler:
>
> Q1. How do we build the pPOD actors against Kepler 1.0 rather than
> against the current revision in the repository?
>
> Q2. Where should we put the pPOD actors and support classes in the
> Kepler repository directory structure? If we put them in, say, src/
> org/ppod (under kepler), then all of this code will be seen by the
> build system and all kepler developers will find themselves
> building code that is not necessarily meant to be compiled against
> the current revision of Kepler in the repository. If any classes
> the pPOD package depends on elsewhere in Kepler are renamed, moved,
> removed, or changed significantly, the pPOD code will no longer
> build against the current revision of Kepler in the repository.
> The build will break.
>
> A further complication is that once the 1.0-compatible version of
> the pPOD package is out, we likely will want to begin building
> against the latest revision of Kepler in the repository until the
> next version of Kepler is out, at which point we'll want to build
> against *that* supported version of Kepler.
>
> In short, we need to be able to choose which revision of Kepler in
> the repository we build the pPOD package against, without
> disturbing folks working with the current revision of Kepler.
>
>
> *** Solutions? ***
>
> S1. On the surface this problem sounds like one that could be
> solved with branches. And I believe it possibly could, and that
> branches likely will be involved in some way (for example, we will
> still want to provide bug fixes to the 1.0-compatible version of
> the pPOD actors when we're working on the 1.1-compatible release).
>
> However, is this how we want to treat the Kepler 1.0 release branch
> in the repository? Do we want to be checking hundreds of source
> files into the Kepler 1.0 release branch indefinitely, and almost
> certainly introducing new bugs and instabilities there?
>
> S2. An alternative would be to create a separate source tree for
> pPOD in the svn repository (as a peer to kepler, for example). The
> build system would not include this 'extension' when compiling the
> Kepler java sources. Instead, we might build the pPOD package
> against the Kepler 1.0 release jar itself. This approach might
> scale better and enable everyone providing their users 1.0-
> compatible actors to share their work with the community through
> the Kepler repository without requiring that their code work with
> the current revision of Kepler.
>
> The problem with this second approach is that it might not be so
> easy to build against the current revision of Kepler, should one
> choose to do so (and as we likely will between releases of the pPOD
> package). At least, not without building the Kepler jar each time.
>
> S3. A variant of S2 would be to enable developers to tell the
> build system to optionally include extensions in the build. With
> the extensions stored as peers to the kepler directory in the
> repository, developers could choose which extensions to check out,
> and specify different branches or tags for Kepler and for each
> extension checked out.
>
> This final approach is the one we currently use at UCD. We store
> our Kepler extensions in our own repository at UCD and use a
> tweaked version of Kepler's standard build.xml file that supports
> extensions in this way. One nice feature of this approach is that
> it does not require all extensions to be stored in the same
> repository. The problem with our current approach is that it is
> harder for us to share our extensions with the rest of the
> community. We'd much rather use the Kepler repository, but at the
> moment we can't as explained above. In short...
>
>
> *** Help! ****
>
> Does anyone else have similar needs? Does anyone have any thoughts
> or advice? Are there other solutions? The UCD team needs a
> solution soon because we want to share our source code widely with
> others in the community, *and* we need to make a release of the
> pPOD actors (as well as a number of other extensions we're
> currently working on outside the Kepler repository for the same
> reasons) to users (including people working here at the UCD Genome
> Center) in the very near future (a few weeks from now).
>
> Thanks very much for your thoughts and help!
>
> Tim
>
>
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at ecoinformatics.org
> http://mercury.nceas.ucsb.edu/ecoinformatics/mailman/listinfo/
> kepler-dev
>
>
>
>
> --
> Tristan King
> Research Officer,
> eResearch Centre
> James Cook University, Townsville Qld 4811
> Australia
>
> Phone: +61747816902
> E-mail: tristan.king at jcu.edu.au www: http://eresearch.jcu.edu.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mercury.nceas.ucsb.edu/ecoinformatics/pipermail/kepler-dev/attachments/20080626/ce0551dc/attachment-0001.htm
More information about the Kepler-dev
mailing list