[kepler-dev] KARs and module dependencies

[Derik Barseghian]

Thanks very much Ben and Dan for the discussions on this topic.

What Ben lists is essentially my plan, and I'm moving forward with  
implementation. To summarize some additional points:

- This plan requires module repositories maintain all published  
versions of a module.
I will:
- Include the complete "currently active" module list (with version  
numbers) in the module-dependencies KAR attribute.
- Eliminate the dependsOnModule KAR entry attribute, since it will no  
longer be used, updating the KAR version, schema, etc in the process.
- Remove any code that was inserting a dependsOnModules value into the  
workflow.
- Create a new user preference for the strictness of KAR compliance.  
I'm still determining the different modes. But e.g. if the user is set  
to 'very strict', in order to open a KAR, they will be prompted to  
import and restart with the exact same module set (with matching  
versions) as that which created the KAR. Essentially the more strict  
this setting, the more warnings a user will get when trying to open  
KARs.

Derik


On Jul 22, 2010, at 4:35 PM, ben leinfelder wrote:

> After bouncing these ideas around with Derik, here's a hybrid  
> approach to handling module dependencies:
>
> -begin including module version number when writing module  
> dependencies in a KAR file
> -when opening [possibly older] KARs that require non-vanilla modules:
> 	-if the module version is not specified, then fetch the latest  
> release of that module
> 	-if the module version is specified, then fetch that version*
> *In practice we'll probably want Strict and Lax modes so that we  
> aren't constantly swapping out modules each time we open a different  
> KAR [for minor version changes].
>
> Additional notes:
> -Development on the trunk - where there is no module version -  
> should also be considered a special case that does not trigger  
> module download. It'd be nice to resolve the dependency using  
> modules from the trunk if we are running from it.
> -We don't want to inadvertently downgrade someone who is opening an  
> older KAR but wants to work with a newer version of the module. We  
> need to be able to save an older KAR with newer module features.
> -We do want to allow a downgrade in cases where old features need to  
> be used, say when reproducing workflow run results from a KAR - the  
> whole point is that we reproduce them exactly - "archive" being the  
> operative word.
>
> Hope I captured most of our discussion!
> -ben
>
> On Jul 22, 2010, at 2:34 PM, Derik Barseghian wrote:
>
>> Hi all,
>>
>> I've found a bug that's blocking the reporting-2.0 release: when  
>> you're in vanilla kepler, and have a KAR in MyWorkflows that was  
>> created with modules you don't have installed, the KAR 'Import  
>> Dependent Modules' context menu item doesn't work: http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5099
>> I can take this bug if Kepler/CORE members can clarify what the  
>> desired behavior is -- a significant design decision is made  
>> depending on how this is fixed.
>>
>> The issue is basically that in a KAR manifest there are module- 
>> dependencies and dependsOnModules attributes, but the values stored  
>> are module names without version number. The 'Import Dependent  
>> Modules' action attempts to use these values to download  
>> dependencies, but fails because it tries to e.g. fetch  
>> provenance.zip instead of provenance-2.0.0.zip.
>>
>> Two issues come to mind:
>>
>> 1) When created, should a KAR manifest store exact versions of  
>> module dependencies instead of just module names?
>>
>> 2) Which versions of modules should the 'Import Dependent Modules'  
>> attempt to fetch and install?
>>
>> A) If we start storing module versions in the manifest, exactly  
>> those? This will not work if a module version is no longer  
>> available, so we would likely want to keep all old versions of  
>> modules in the released area of our repository (is this already the  
>> plan?).
>>
>> B) The newest? This means requiring modules remain backward  
>> compatible with all versions of their KAR artifacts. This would  
>> also require fetching the modules in the proper order. If  
>> aSuite-2.1.0 is out and it depends on bSuite-2.0.0, and  
>> bSuite-2.1.0 is also available, bSuite-2.1.0 should not be  
>> downloaded. If the manifest module-dependencies attribute doesn't  
>> store these in the right order (not sure), given a list of modules  
>> to download, the module manager code would have to be able to  
>> figure this out (maybe it already can?).
>>
>> C) Something else? :)
>>
>> I think B may be the way to go, and also does not require we do 1),  
>> even though we may want to do that in the future.
>>
>> Thanks,
>> Derik
>> _______________________________________________
>> Kepler-dev mailing list
>> Kepler-dev at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>