[kepler-dev] KARs and module dependencies

[ben leinfelder]

After bouncing these ideas around with Derik, here's a hybrid approach to handling module dependencies:

-begin including module version number when writing module dependencies in a KAR file
-when opening [possibly older] KARs that require non-vanilla modules:
	-if the module version is not specified, then fetch the latest release of that module
	-if the module version is specified, then fetch that version*
*In practice we'll probably want Strict and Lax modes so that we aren't constantly swapping out modules each time we open a different KAR [for minor version changes]. 

Additional notes:
-Development on the trunk - where there is no module version - should also be considered a special case that does not trigger module download. It'd be nice to resolve the dependency using modules from the trunk if we are running from it.
-We don't want to inadvertently downgrade someone who is opening an older KAR but wants to work with a newer version of the module. We need to be able to save an older KAR with newer module features.
-We do want to allow a downgrade in cases where old features need to be used, say when reproducing workflow run results from a KAR - the whole point is that we reproduce them exactly - "archive" being the operative word.

Hope I captured most of our discussion!
-ben

On Jul 22, 2010, at 2:34 PM, Derik Barseghian wrote:

> Hi all,
> 
> I've found a bug that's blocking the reporting-2.0 release: when you're in vanilla kepler, and have a KAR in MyWorkflows that was created with modules you don't have installed, the KAR 'Import Dependent Modules' context menu item doesn't work: http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5099
> I can take this bug if Kepler/CORE members can clarify what the desired behavior is -- a significant design decision is made depending on how this is fixed.
> 
> The issue is basically that in a KAR manifest there are module-dependencies and dependsOnModules attributes, but the values stored are module names without version number. The 'Import Dependent Modules' action attempts to use these values to download dependencies, but fails because it tries to e.g. fetch provenance.zip instead of provenance-2.0.0.zip.
> 
> Two issues come to mind:
> 
> 1) When created, should a KAR manifest store exact versions of module dependencies instead of just module names?
> 
> 2) Which versions of modules should the 'Import Dependent Modules' attempt to fetch and install?
> 
> A) If we start storing module versions in the manifest, exactly those? This will not work if a module version is no longer available, so we would likely want to keep all old versions of modules in the released area of our repository (is this already the plan?).
> 
> B) The newest? This means requiring modules remain backward compatible with all versions of their KAR artifacts. This would also require fetching the modules in the proper order. If aSuite-2.1.0 is out and it depends on bSuite-2.0.0, and bSuite-2.1.0 is also available, bSuite-2.1.0 should not be downloaded. If the manifest module-dependencies attribute doesn't store these in the right order (not sure), given a list of modules to download, the module manager code would have to be able to figure this out (maybe it already can?).
> 
> C) Something else? :)
> 
> I think B may be the way to go, and also does not require we do 1), even though we may want to do that in the future.
> 
> Thanks,
> Derik
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev