[kepler-dev] Proposed Use Cases for New Build System
cxh at eecs.berkeley.edu
Mon Apr 14 11:48:30 PDT 2008
Below are some comments. Pretty much everything looks good, except
I'd like to discuss forking Ptolemy.
> Compile Kepler including necessary portions of Ptolemy, which
> will now be checked into our repository, to be frozen except
> when updated periodically by management. No more working at
> the head of Ptolemy, except at own risk and through extension
This has the upside of not being victim to Ptolemy breaking things.
It has the downside of not getting fixes and updates. Basically,
this is forking the Ptolemy tree. Usually forking is done when
people can't agree. Further discussion would be good before going
down this route.
If the primary issue is the UI, then perhaps working on a new UI
that allows easier customization would be good.
I've been fairly happy with how the Kepler/Ptolemy interaction has
worked out, but then my view is only one side of things.
> Compile differerent versions of Kepler, based on SCM
> tags. Interact with the SCM system to download desired version
> of Kepler to location relative to the build file at build time
> if not already present.
In my experience, the only people who don't work at the head are those
building releases. Usually providing tarballs is a more effective way
to build old releases. One issue is that getting the source code and
the jar files for a particular version can be tricky unless everything
is in SCM However, effectively using SCM tags is a good idea. BTW -
is this the time to move to SVN?
> Explicitly support extensions to Kepler, which can overwrite
> files in Kepler itself, thus giving extension developers full
> control. All extensions go into extension folders relative to
> Kepler. A priority mechanism is present in case multiple
> extensions are used, so that files in Extension A can be
> specified to take priority Extension B. All extensions have a
> higher priority than Kepler by default.
The java classpath mechanism provides this functionality? Probably
the easiest thing is to make sure that $CLASSPATH is used before
the Kepler/Core classes. We tried this, and it gets rough because
if someone has an older version of a product in their CLASSPATH
and then starts up a newer version, the older version is what is run.
Perhaps using a separate path (other than CLASSPATH) is an idea?
Thinking about how to handle multiple versions of Kepler and their
extensions might be useful. Individiual Java JDK releases use individual
jre/lib/ext directories. The downside is that I end up copying
the extensions I use around for each upgrade. I think the JDK
has endorsed directories, which could help.
> Environmental variables are removed from Kepler. Kepler will
> be able to detect where it is running at runtime and will know
> where Ptolemy code is relative to itself, and behave
> accordingly. No environmental variables mean increased
> cross-platform ease of use and enable the use of multiple IDEs
> much more easily.
Yay! We tried to keep use of PTII out of Ptolemy, but ended up
needing it in a few places, usually to find files that were then
passed to external programs. Even these cases can usually be worked
on. One trick is to look for a class, say NamedObj.class in the
classpath and assume that PTII is two levels up. This fails because
if NamedObj.class is in a jar file as part of WebStart, then the
location of the jar file odd and the two levels up will not work.
ptolemy.util.StringUtilities.getProperty() is a wrapper for
System.getProperty() that tries to figure out the setting for $PTII.
Perhaps something similar could be used by Kepler?
One possible workaround is to build a properties file at configuration
time that includes locations of various tools. Perhaps it would
be possible to rerun configure from the tool and generate a new
I'm not sure if it is possible to get away from environment variables,
but it is worth a try.
> No special build process for Eco Grid jars or any other jars.
> Build all kar files. Kar files are no longer recompiled unless
> they are touched. Building kars is less spammy.
> All third party jars are stored in repositories seperate from
> the source code. Transitive dependencies between jars are
> managed, such that if you specify that Jar A is necessary, and
> Jar A depends on Jar B, and Jar B depends on Jar C, all you
> have to do is specify that Jar A is necessary, and Jar B and
> Jar C are retrieved automatically.
So to build I would need to check out two repositories? I'm not sure
if this is wise. What are the benefits of having two repositories?
> All required jars, including extension jars, are downloaded
> into one directory, to allow easy use of multiple IDEs.
Ok, this could be an advantage of two repositories. I'm still not
totally convinced . . .
> Manage dependencies on, including the version priority, within
> build system.
> Package compiled code and resources into a jar.
I've spent quite a bit of time on the Ptolemy build system so that
code is modular in jar files. This is why Ptolemy uses makefiles -
so that we have a manifest for what should be in each jar file,
including subjars. One issue is that absolutely noone cares about
this other than the release engineer. Automated tools that run
different configurations help find dependencies.
> Create build time report specifying all third-party jars used,
> including transitive dependencies so that management decisions
> can be made regarding those jars more easily.
Nice. Who will look at this and how often?
> Developers are not allowed to create _core_ third-party
> dependencies of any kind without management approval to ensure
> jar compatability and that licensing requirements are
> met. Developers may use third-party jars without management
> approval by using the extension mechanism, but the core may
> not depend on such jars. There will be a management task to
> check for compliance, and automatically flag any developer
> that creates an unauthorized dependency within the core.
The above is key. Clearly defining the core is critical.
Being able to build a version of Kepler that includes just the Core
> Automatically perform code management tasks, such as fixing
> import statements and ensuring proper code formatting, from
> the build system by checking out code from the SCM,
> automatically altering it to ensure it conforms to management
> specifications, and checking it back in.
This would be wonderful. I suggest using Eclipse for formatting etc.
I usually do this by hand. I suggest deciding a style that is
enforced by a tool and sticking with it.
> All code within the core shall be in the parent package
> org.kepler.core. A task to automatically promote all code in a
> specified directory to the core shall automatically refactor
> such code and add it to the core. Third-party dependencies
> must be added manually.
> Running Ptolemy alone is no longer supported.
> Run Kepler. -kepler is the default, and while it can be
> overridden, it no longer need be specified by the build, at
> the command line, or by any IDE. This will allow easier use of
> multiple IDEs.
> Run Kepler with a default workflow environment.
> Run MomlExecute to run a workflow from the command-line with
> no Swing components. This will be renamed to something more
> Run PtExecute to run a workflow from the command-line with
> Swing components. This will be renamed to something more
The above all sound good
> Run a set of unit tests.
> Run one unit tests.
> Test workflows.
The copyright about: facility allows a tester to open up all the
models listed in a html file. From there, the user can run them.
Key workflows should have system tests that take the output and
compare it against a known output.
You'll also need code coverage tools and reports.
> Create documentation
Further amplification is needed here?
> Create only one Windows batch file and one Unix script file to
> run Kepler normally or from the command line in all modes.
Ptolemy does this by having a shell script that then gets symbolically
linked from different names. The Windows batch is also generated
from this script.
> Create tar.gz and zip source code distributions versions for
> non-affiliated developers not familiar with anonymous access
> to the SCM system. Also, to create distributions that
> correspond to particular releases for easy access in the
Creating these every night and running smoke tests on them would
BTW - an interesting issue came up where when I was building the
Izpack installer that includes the source code. The source code
was from Windows, so it had \r\n line endings. This really did
not work under the Mac.
> Build Windows and Mac executables.
> Java Web Start is no longer supported.
BTW - I find that building Java Web Start is a good way to sort out
dependency issues for packaging because it is much faster to build Web
Start, find a bug, rebuild the jar file and rerun it than it is to
build an installer and re-run it. However, if the installer can
optionally use just jar files, then there is less of an advantage to
Web Start. Basically, I do the Web Start deployment first, then
the Izpack deployment, which gives me both with very little work.
If you lose WebStart, then you lose a way to easily deploy models
in Grid Computing etc.
Still, de-emphasizing WebStart is probably a good idea.
> Kepler WAR files are no longer supported.
> Run Kepler repository and add kar files to
> repository. No longer supported, as we are redesigning
> Run QueryBuilder is no longer supported. (?)
> Run DataCacheViewer is no longer supported. (?)
> Retrieve EML files is no longer supported. (?)
> Find Bugs. May consider as a management task. (?)
Here do you mean the FindBugs program? I've spent quite a bit
of time cleaning up problems in the Ptolemy code. It is mainly an
exercise, but it does result in cleaner code. I recommend having
developers run FindBugs and DocCheck on their code before considering
it shippable. Addressing the Eclipse warnings should also be done.
I recently added the following checklist to the end of the Ptolemy
II style guide. I encourage Kepler to have a similar checklist.
Below is a checklist for common issues with new Ptolemy II files.
6.6.1 File Structure
0 Copyright - Does the file have the copyright? (section 6.2.1)
0 Is the copyright year correct? New files should have the just the
current year (section 6.2.1)
6.6.2 Class comment
0 Is the first sentence of the class comment a cogent and complete
summary? (section 6.3)
0 Are these tags present (section 6.3.2):
0 Are the constructors, methods and variables separated by the
appropriate comment lines? (section 6.2
6.6.3 Constructor, method, field and inner class Javadoc documentation.
0 Within each section, is each Javadoc comment alphabetized?
0 Is the first sentence a cogent and complete summary in the
imperative case? (section 6.3.5)
0 Are all the parameters of each method clearly documented?
0 Are the descriptions of each exception useful?
0 Did you run doccheck and review the results? See
http://chess.eecs.berkeley.edu/ptexternal/nightly (section 6.3.1)
0 Did you run spell check on the program and fix errors? (section
0 Did you run format the file using Eclipse? (section 6.1)
0 Did you fix the imports using Eclipse? (section 6.2.2)
More information about the Kepler-dev