[kepler-dev] Meeting about .kepler and related issues
Timothy McPhillips
tmcphillips at mac.com
Thu Jun 4 13:21:49 PDT 2009
David and I have started putting together a little table classifying
the various kinds of information stored by Kepler (on the scientist's
computer) so that we can evaluate proposals for where to store and how
to organize this information. Can we use and update this table during
our discussion at 2 PM?
https://kepler-project.org/developers/teams/framework/classification-of-persistent-system-state
Cheers,
Tim
On Jun 3, 2009, at 10:44 AM, Chad Berkley wrote:
> works for me.
>
> David Welker wrote:
>> Matt, Tim, Chad, Aaron and other interested parties:
>> Would you be available to meet on Thursday at 2 pm to discuss the
>> issues identified below?
>> David
>>>
>>> Feel free to set up a meeting.
>>>
>>> David Welker wrote:
>>>> A couple of points:
>>>>
>>>> (1) The question of whether what support we are going to have for
>>>> multiple installations of Kepler versus having one installation
>>>> of Kepler that is configurable has yet to be fully developed via
>>>> discussion. Your design assumes that we are supporting multiple
>>>> installations of Kepler on the same machine and that each of
>>>> these multiple installations have their own cache. Those are all
>>>> reasonable ideas, but these ideas have yet to be discussed and
>>>> agreed upon by Kepler management or the various development
>>>> teams. This is not a minor implementation decision, and it has
>>>> implications that need to be discussed.
>>>>
>>>> (2) The implications of the fact that Kepler could be installed
>>>> in a read-only area are far reaching. It certainly affects the
>>>> build system. Right now, modules.txt in the build-area is
>>>> regularly rewritten with every issuance of the change-to command.
>>>> But, what if Kepler is installed in a read-only area? It sounds
>>>> like in that case, we are going to have to think about what we
>>>> can do if modules.txt cannot be written. Like, for example,
>>>> having a version of modules.txt in the .kepler directory that is
>>>> read from for example. So, this is definitely not unrelated to
>>>> the general work you are doing with .kepler right now. Also, we
>>>> need to think about where kars are build. Originally, kars were
>>>> built in the common module. Then, they were built in the
>>>> kepler.modules directory. Then, they were build in the module
>>>> where the came from. However, if Kepler is installed in a read-
>>>> only locations, none of these solutions for where kars are built
>>>> would be acceptable. We would probably want to build kars in
>>>> the .kepler or similar directory in the user directory.
>>>>
>>>> (3) There are also implications here for the configuration
>>>> system. In general, the system default configurations could be in
>>>> a read-only area. Therefore, user changes to configuration need
>>>> to be stored in a different location than the original
>>>> configuration files, which would contain default configuration
>>>> options.
>>>>
>>>> (4) For development purposes, I do not think we need to support
>>>> caches for multiple installations of Kepler. However, for your
>>>> testing purposes, it would be acceptable to have the build system
>>>> generate these sorts of files, just as the installer would. But,
>>>> before that, we need to think about what sort of support we are
>>>> going to provide for multiple installations of Kepler and only
>>>> then go forward with an implementation or strategy.
>>>>
>>>> (5) Kepler itself should not try to write files that in the
>>>> normal course of events should be written by the installer only.
>>>> Instead, the system should have a sensible default behavior that
>>>> it utilizes when such files are absent.
>>>>
>>>> (6) If you delete "only one file" in an installation of Kepler or
>>>> any other software, there is a good chance that this will in fact
>>>> mess up the software. Of course, the more robust we can make the
>>>> system, the better. However, there is no reason to think that
>>>> people will be likely to delete this file. Especially if it is
>>>> written to a less visible location in a subdirectory of the build-
>>>> area folder. Of course, I must admit that when I see a file like
>>>> this in kepler.modules, I am in fact tempted to delete it because
>>>> it doesn't feel like it belongs there.
>>>>
>>>> (7) In general, we should avoid storing non module files in
>>>> kepler.modules to reduce the probability of name clashes.
>>>>
>>>> (8) Also, if the purpose of the class is to uniquely identify a
>>>> particular installation of Kepler, I think a better name than
>>>> InstanceAuthNamespace should be selected. It is not at all clear
>>>> from the file names what the function of this file is -- although
>>>> one gets a sense of how it does whatever it does do (i.e. with
>>>> namespaces). I think in general, it would be better to select
>>>> file names thinking about "what" rather than "how," especially as
>>>> implementations are liable to change in the future.
>>>>
>>>> Anyway, I think we need to have a discussion about these issues
>>>> that includes Tim and Matt and other interested parties at a
>>>> minimum, and not just you, me, and Chad. We really need to think
>>>> about the implications of these decisions before moving forward.
>>>>
>>>> David
>>>>
>>>>>
>>>>> I think it's ok to have that file in the project root, alongside
>>>>> all of the read-only module jars, the installer could initially
>>>>> generate the file, but we still need the runtime mechanism for
>>>>> development purposes. It also increases the reliability quite a
>>>>> bit, imagine if I had to rerun my installer just because I
>>>>> deleted that one file...
>>>>>
>>>>> Aaron
>>>>>
>>>>>
>>>>> David Welker wrote:
>>>>>> If you are going to have a file that uniquely identifies an
>>>>>> installation and that file is going to be written to
>>>>>> kepler.modules (I think a better location would be in a
>>>>>> subfolder of the build-area folder where it not likely to be
>>>>>> seen very often), then it must be written by the installer and
>>>>>> not by any other process. In general, Kepler is likely to be
>>>>>> installed in a read-only area of disk, so such a file must be
>>>>>> generated only once at installation time.
>>>>>>
>>>>>> David
>>>>>>>
>>>>>>> I thought that would work (and actually moved it there
>>>>>>> yesterday) but the problem arises that if you have two
>>>>>>> different installations of Kepler on the same machine they
>>>>>>> both end up using the .kepler/InstanceAuthNamespace file and
>>>>>>> therefore the same cache. Now this may be what you want but
>>>>>>> may not be what you want. Say I have a kepler base
>>>>>>> configuration installation and a Kepler WRP installation,
>>>>>>> there are cases where I may want them to use the same cache or
>>>>>>> I may not want to use the same cache. If the
>>>>>>> InstanceAuthNamespace is in .kepler then they both MUST use
>>>>>>> the same cache. If the InstanceAuthNamespace file is stored
>>>>>>> at the project root then they can both use different caches or
>>>>>>> they can both use the same cache (by copying the
>>>>>>> InstanceAuthNamespace file to both project root directories).
>>>>>>>
>>>>>>> Aaron
>>>>>>>
>>>>>>> Chad Berkley wrote:
>>>>>>>> Could it go in the root of the .kepler directory so the
>>>>>>>> system will know where to find it? Or maybe in some 'common'
>>>>>>>> directory in .kepler that doesn't rely on the unique id system?
>>>>>>>>
>>>>>>>> chad
>>>>>>>>
>>>>>>>>
>>>>>>>> Aaron Schultz wrote:
>>>>>>>>>
>>>>>>>>> Hi Chad,
>>>>>>>>>
>>>>>>>>> The InstanceAuthNamespace file contains a java serialized
>>>>>>>>> string for the unique authority and namespace of any given
>>>>>>>>> kepler installation (which is either retrieved from a
>>>>>>>>> webservice or assigned via UUID).
>>>>>>>>>
>>>>>>>>> For each installation of kepler on a given machine there
>>>>>>>>> would be (actually there now is) a subdirectory in
>>>>>>>>> the .kepler directory that was named using the authority and
>>>>>>>>> namespace.
>>>>>>>>> Each of these .kepler/instance directories would contain
>>>>>>>>> it's own cache as we discussed yesterday....
>>>>>>>>>
>>>>>>>>> So we need that one file to find all the other files for any
>>>>>>>>> given installation of kepler...
>>>>>>>>>
>>>>>>>>> Aaron
>>>>>>>>>
>>>>>>>>> Chad Berkley wrote:
>>>>>>>>>> Hi Aaron,
>>>>>>>>>>
>>>>>>>>>> I've been seeing this file appear and wondering what it
>>>>>>>>>> was. I think things like this should be written to .kepler
>>>>>>>>>> or some other user directory. I agree with David that the
>>>>>>>>>> kepler project directory should probably not be written to
>>>>>>>>>> in general. Maybe this type of file should be written to
>>>>>>>>>> the cache so that it could be programatically purged if and
>>>>>>>>>> when it needs to be.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> chad
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> David Welker wrote:
>>>>>>>>>>> Hi Aaron,
>>>>>>>>>>>
>>>>>>>>>>> In general, you need to avoid writing anything to the
>>>>>>>>>>> kepler.modules project root directory. In the general
>>>>>>>>>>> case, this directory is likely to be stored in a read-only
>>>>>>>>>>> area of disk.
>>>>>>>>>>>
>>>>>>>>>>> David
>>>>>>>>>>>>
>>>>>>>>>>>> You will want to delete the InstanceAuthNamespace file in
>>>>>>>>>>>> your project root directory.
>>>>>>>>>>>>
>>>>>>>>>>>> Otherwise you will have troubles getting proper object
>>>>>>>>>>>> ids for LSIDs.
>>>>>>>>>>>>
>>>>>>>>>>>> Aaron
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Kepler-dev mailing list
>>>>>>>>>>>> Kepler-dev at kepler-project.org
>>>>>>>>>>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Kepler-dev mailing list
>>>>>>>>>>> Kepler-dev at kepler-project.org
>>>>>>>>>>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>> _______________________________________________
>> Kepler-dev mailing list
>> Kepler-dev at kepler-project.org
>> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
> _______________________________________________
> Kepler-dev mailing list
> Kepler-dev at kepler-project.org
> http://mercury.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-dev
More information about the Kepler-dev
mailing list