[kepler-dev] Meeting about .kepler and related issues

[Aaron Schultz]

Sure sounds good, 2pm to 3pm is fine for me.  I will have to leave at 3pm.

Any solution is fine with me as long as each module of each Kepler 
installation has a place on disk that it can write to that does not 
clash with other modules from other installations.

Whether that solution is creating a subdirectory in the project root 
folder for modules to write to
Storing the path to a location on disk that a module can write to (as 
the InstanceAuthNamespace file is essentially doing now)
or having a database where each kepler installation connects to and 
selects the directory name that it is meant to use for writing to
or some other solution....

whether or not the cache is stored in that directory, just depends on 
what you want your cache to be doing,

is it really the library where you want to keep information that exists 
nowhere else?
or is it just a useful tool for making the program run faster?
If it is just there to make things faster then it is simply a reflection 
of what is on disk
and therefore can be rebuilt and deleted without effort (eliminating 
duplication)

regardless of the cache, each module in each kepler installation needs a 
unique location on disk it can use for writing and reading that is not 
inside it's own module directory (since this will someday be jarred and 
become read-only through the publication process - from how I understand 
that it will work now).

In eclipse the writeable directory for modules (aka bundles) is the  
"eclipse/configuration/<bundle name>/" directory
the modules (aka bundles) themselves are stored in
"eclipse/plugins/<bundle name>.jar"
where only the jar is a read-only entity and preferably signed with a 
security certificate

I would point out that the core configuration file for an eclipse 
instance does not reside inside a module but inside the project root folder
then each bundle can contribute configuration information to the 
application as a whole.

I see no reason why we cannot write things to the project root directory 
at the base system level.

Aaron


Matt Jones wrote:
> Hi All,
>
> I can not make a meeting this week, but I will be very interested in
> the proposal that will come out of this discussion, and will comment
> on it when I see a concrete proposal.  In the meantime, I'd second the
> idea that we should not be writing new or temporary files to the
> kepler installation directories -- any temporary or instance-specific
> files that need to be preserved across upgrades probably should reside
> in .kepler.  I don't fully understand what is being proposed yet, but
> with my current limited perspective I can see some issues with having
> multiple subdirectories within .kepler for each version -- for one
> thing it would represent a lot of unnecessary duplication.  For
> example, there would be multiple copies of the cache database and
> cached data files, as two examples. Looking forward to seeing your
> proposal when I return next week.
>
> Matt
>
> On Tue, Jun 2, 2009 at 3:52 PM, David Welker <david.v.welker at gmail.com> wrote:
>   
>> Matt, Tim, Chad, Aaron and other interested parties:
>>
>> Would you be available to meet on Thursday at 2 pm to discuss the issues
>> identified below?
>>
>> David
>>     
>>> Feel free to set up a meeting.
>>>
>>> David Welker wrote:
>>>       
>>>> A couple of points:
>>>>
>>>> (1) The question of whether what support we are going to have for
>>>> multiple installations of Kepler versus having one installation of Kepler
>>>> that is configurable has yet to be fully developed via discussion. Your
>>>> design assumes that we are supporting multiple installations of Kepler on
>>>> the same machine and that each of these multiple installations have their
>>>> own cache. Those are all reasonable ideas, but these ideas have yet to be
>>>> discussed and agreed upon by Kepler management or the various development
>>>> teams. This is not a minor implementation decision, and it has implications
>>>> that need to be discussed.
>>>>
>>>> (2) The implications of the fact that Kepler could be installed in a
>>>> read-only area are far reaching. It certainly affects the build system.
>>>> Right now, modules.txt in the build-area is regularly rewritten with every
>>>> issuance of the change-to command. But, what if Kepler is installed in a
>>>> read-only area? It sounds like in that case, we are going to have to think
>>>> about what we can do if modules.txt cannot be written. Like, for example,
>>>> having a version of modules.txt in the .kepler directory that is read from
>>>> for example. So, this is definitely not unrelated to the general work you
>>>> are doing with .kepler right now. Also, we need to think about where kars
>>>> are build. Originally, kars were built in the common module. Then, they were
>>>> built in the kepler.modules directory. Then, they were build in the module
>>>> where the came from. However, if Kepler is installed in a read-only
>>>> locations, none of these solutions for where kars are built would be
>>>> acceptable. We would probably want to build kars in the .kepler or similar
>>>> directory in the user directory.
>>>>
>>>> (3) There are also implications here for the configuration system. In
>>>> general, the system default configurations could be in a read-only area.
>>>> Therefore, user changes to configuration need to be stored in a different
>>>> location than the original configuration files, which would contain default
>>>> configuration options.
>>>>
>>>> (4) For development purposes, I do not think we need to support caches
>>>> for multiple installations of Kepler. However, for your testing purposes, it
>>>> would be acceptable to have the build system generate these sorts of files,
>>>> just as the installer would. But, before that, we need to think about what
>>>> sort of support we are going to provide for multiple installations of Kepler
>>>> and only then go forward with an implementation or strategy.
>>>>
>>>> (5) Kepler itself should not try to write files that in the normal course
>>>> of events should be written by the installer only. Instead, the system
>>>> should have a sensible default behavior that it utilizes when such files are
>>>> absent.
>>>>
>>>> (6) If you delete "only one file" in an installation of Kepler or any
>>>> other software, there is a good chance that this will in fact mess up the
>>>> software. Of course, the more robust we can make the system, the better.
>>>> However, there is no reason to think that people will be likely to delete
>>>> this file. Especially if it is written to a less visible location in a
>>>> subdirectory of the build-area folder. Of course, I must admit that when I
>>>> see a file like this in kepler.modules, I am in fact tempted to delete it
>>>> because it doesn't feel like it belongs there.
>>>>
>>>> (7) In general, we should avoid storing non module files in
>>>> kepler.modules to reduce the probability of name clashes.
>>>>
>>>> (8) Also, if the purpose of the class is to uniquely identify a
>>>> particular installation of Kepler, I think a better name than
>>>> InstanceAuthNamespace should be selected. It is not at all clear from the
>>>> file names what the function of this file is -- although one gets a sense of
>>>> how it does whatever it does do (i.e. with namespaces). I think in general,
>>>> it would be better to select file names thinking about "what" rather than
>>>> "how," especially as implementations are liable to change in the future.
>>>>
>>>> Anyway, I think we need to have a discussion about these issues that
>>>> includes Tim and Matt and other interested parties at a minimum, and not
>>>> just you, me, and Chad. We really need to think about the implications of
>>>> these decisions before moving forward.
>>>>
>>>> David
>>>>
>>>>         
>>>>> I think it's ok to have that file in the project root, alongside all of
>>>>> the read-only module jars, the installer could initially generate the file,
>>>>> but we still need the runtime mechanism for development purposes.  It also
>>>>> increases the reliability quite a bit, imagine if I had to rerun my
>>>>> installer just because I deleted that one file...
>>>>>
>>>>> Aaron
>>>>>
>>>>>
>>>>> David Welker wrote:
>>>>>           
>>>>>> If you are going to have a file that uniquely identifies an
>>>>>> installation and that file is going to be written to kepler.modules (I think
>>>>>> a better location would be in a subfolder of the build-area folder where it
>>>>>> not likely to be seen very often), then it must be written by the installer
>>>>>> and not by any other process. In general, Kepler is likely to be installed
>>>>>> in a read-only area of disk, so such a file must be generated only once at
>>>>>> installation time.
>>>>>>
>>>>>> David
>>>>>>             
>>>>>>> I thought that would work (and actually moved it there yesterday) but
>>>>>>> the problem arises that if you have two different installations of Kepler on
>>>>>>> the same machine they both end up using the .kepler/InstanceAuthNamespace
>>>>>>> file and therefore the same cache.  Now this may be what you want but may
>>>>>>> not be what you want.  Say I have a kepler base configuration installation
>>>>>>> and a Kepler WRP installation, there are cases where I may want them to use
>>>>>>> the same cache or I may not want to use the same cache.  If the
>>>>>>> InstanceAuthNamespace is in .kepler then they both MUST use the same cache.
>>>>>>>  If the InstanceAuthNamespace file is stored at the project root then they
>>>>>>> can both use different caches or they can both use the same cache (by
>>>>>>> copying the InstanceAuthNamespace file to both project root directories).
>>>>>>>
>>>>>>> Aaron
>>>>>>>
>>>>>>> Chad Berkley wrote:
>>>>>>>               
>>>>>>>> Could it go in the root of the .kepler directory so the system will
>>>>>>>> know where to find it?  Or maybe in some 'common' directory in .kepler that
>>>>>>>> doesn't rely on the unique id system?
>>>>>>>>
>>>>>>>> chad
>>>>>>>>
>>>>>>>>
>>>>>>>> Aaron Schultz wrote:
>>>>>>>>                 
>>>>>>>>> Hi Chad,
>>>>>>>>>
>>>>>>>>> The InstanceAuthNamespace file contains a java serialized string for
>>>>>>>>> the unique authority and namespace of any given kepler installation (which
>>>>>>>>> is either retrieved from a webservice or assigned via UUID).
>>>>>>>>>
>>>>>>>>> For each installation of kepler on a given machine there would be
>>>>>>>>> (actually there now is) a subdirectory in the .kepler directory that was
>>>>>>>>> named using the authority and namespace.
>>>>>>>>> Each of these .kepler/instance directories would contain it's own
>>>>>>>>> cache as we discussed yesterday....
>>>>>>>>>
>>>>>>>>> So we need that one file to find all the other files for any given
>>>>>>>>> installation of kepler...
>>>>>>>>>
>>>>>>>>> Aaron
>>>>>>>>>
>>>>>>>>> Chad Berkley wrote:
>>>>>>>>>                   
>>>>>>>>>> Hi Aaron,
>>>>>>>>>>
>>>>>>>>>> I've been seeing this file appear and wondering what it was.  I
>>>>>>>>>> think things like this should be written to .kepler or some other user
>>>>>>>>>> directory.  I agree with David that the kepler project directory should
>>>>>>>>>> probably not be written to in general.  Maybe this type of file should be
>>>>>>>>>> written to the cache so that it could be programatically purged if and when
>>>>>>>>>> it needs to be.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> chad
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> David Welker wrote:
>>>>>>>>>>                     
>>>>>>>>>>> Hi Aaron,
>>>>>>>>>>>
>>>>>>>>>>> In general, you need to avoid writing anything to the
>>>>>>>>>>> kepler.modules project root directory. In the general case, this directory
>>>>>>>>>>> is likely to be stored in a read-only area of disk.
>>>>>>>>>>>
>>>>>>>>>>> David
>>>>>>>>>>>                       
>>>>>>>>>>>> You will want to delete the InstanceAuthNamespace file in your
>>>>>>>>>>>> project root directory.
>>>>>>>>>>>>
>>>>>>>>>>>> Otherwise you will have troubles getting proper object ids for
>>>>>>>>>>>> LSIDs.
>>>>>>>>>>>>
>>>>>>>>>>>> Aaron
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Kepler-dev mailing list
>>>>>>>>>>>> Kepler-dev at kepler-project.org
>>>>>>>>>>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>>>>>>>>>>>
>>>>>>>>>>>>                         
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Kepler-dev mailing list
>>>>>>>>>>> Kepler-dev at kepler-project.org
>>>>>>>>>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>                   
>>>>>>>               
>>>>>           
>>>       
>> _______________________________________________
>> Kepler-dev mailing list
>> Kepler-dev at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>
>>     
>
>
>
>