[kepler-dev] [Bug 4702] KAR to module conversion utility

[bugzilla-daemon at ecoinformatics.org]

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=4702

--- Comment #9 from Aaron <aschultz at nceas.ucsb.edu> 2010-02-12 15:05:34 PST ---
> This is an inappropriate conversation for this forum.

And what is the appropriate forum?

> This issue has already been decided.

And who decided this issue?  The managers didn't care either way as long as the
requirements were met.  Have we met the requirements?  Can a User develop a
module, download it and boot it up in Kepler with a high reliability of
success?

> You have already had the opportunity to express your views and you failed to persuade others.

I failed to persuade you, you mean.  And tell me, have you ever built any OSGi
modules?  You certainly hadn't when I was proposing OSGi.  You spent 4 months
of development on the build system before I ever started evaluating OSGi as a
solution and then you fought it vigorously because you had already decided to
develop everything yourself from scratch.  When we were supposed to be doing
design work you were developing.  You never truly evaluated any other
technology and from what I can tell you still haven't (reading articles doesn't
count, go and use the technology).  You chose a solution on your own and you
implemented it.  The managers caved because I gave up!  Adopting a standard is
something we needed to decide as a team, I could not make that decision on my
own.  And quite frankly I couldn't stand to work with you anymore.

> Deal with it. When it comes to Kepler, no one always gets what they want or think best. This definitely includes me.

I have to deal with it every day.

> We had good reasons to take a different approach. We decided against OSGi for many reasons including reasons:

> (1) The complexity that it would foist developers. For example, take this comment from an OSGi-advocate about his experiences trying to implement OSGi at IBM:

<blockquote>
"As you may imagine, one of the most important goals was to hide OSGi as much
as possible. No services, no Import-Package (but Require-Bundle on one
aggregation bundle), basically reduce OSGi to its minimum. Well, the
appreciation was… limited. All people I was working with were exceptional
bright and determined researchers in MLP. They developed highly sophisticated
algorithms on how to analyze unstructured data, but only a few were software
engineers. So the code was… working, but not production ready. Enforcing them
to apply rules that ultimately made it harder for them getting things done and
as a result slowing them down wasn’t something they welcomed very much."
</blockquote>

You should really include references to your quotes, I found it here:
http://osgi.mjahn.net/2009/07/01/osgi-vs-jigsaw-why-cant-we-talk/

Towards the bottom the writer, Mirko Jahn, defines 6 "Properties of a true
module system"
- Isolation
    Our build system does not isolate modules it combines them all together
using the same classloader and a single classpath
- Information Hiding
    In the core module, I have written many public methods that I need in other
classes in core but do not want exposed to other modules.
- Enabling Reuse
    Can someone who creates modules in our module system reuse them elsewhere?
- Predictability
    Overrides in the system causes Unpredictability (btw, you can do overrides
in OSGi so please stop using that argument)
- Flexible binding
    The author claims that no modular system has achieved this.  Has ours?
- Robustness
    You experience this in Eclipse, and every other piece of software you have
ever run, including Kepler.

The fact that you include this quote makes me wonder if you think our software
engineers are not capable of learning OSGi or some other modularization
standard?  I have worked with everyone here and I have developed simple OSGi
applications.  And I can assure you that it is not that hard to do.  Everyone
on our team could come up to speed on OSGi in a couple weeks and be proficient
with it.  The fact that Mirko's team of mostly non-software engineers had
trouble with it is hardly a concern to me.

> Our primary mission at Kepler is to facilitate the development of software that advances the interests of scientists, not slow it down. In the quote above, the OSGi advocate explains the frustrations of trying to implement OSGi within a SINGLE institution. We at Kepler are responsible for creating a platform for MANY institutions. Adding the complexities of OSGi to the development process to achieve a small incremental advantage in hiding the implementation of modules from each other (while sacrificing the ability to do overrides) was discussed and rejected, given our mission. It makes more sense to stick to mainstream Java development rather than have kepler-dev become a place where a primary activity is the fielding a constant barrage of OSGi-related questions.

I am amazed that you have the hubris to make this prediction without knowing
anything about OSGi!  You are convinced that it is so hard because you never
bothered to learn it.  Your lack of understanding was evidenced to me when I
asked about module version ranges in a meeting in October which resulted in
this thread->
https://kepler-project.org/developers/kepler-development-forum/build-and-release-team/274120463#500806490
Do you think many people will take the time to learn how Kepler modules work? 
Do you think we won't get a constant barrage of Kepler module system-related
questions?  Especially when the system can't do what people want or expect?  At
least with OSGi (or whatever other standard that is well used out there) there
are books and mailing lists and blogs and webpages and all kinds of other
sources to help out.

> (2) OSGi may have been developed over 10 years, but it was developed outside of the Java Community Process and the problems it was meant to address are different than the problems we are trying to address. The original context of OSGi was primarily cell phones, where the "modules" tend to be unrelated apps. Also, there is the issue of security, where want to close off your module from others. In our context of scientific development, we want our modules to be more open and there it is both permissible and desirable for one module to modify the functioning of another in the context of a particular suite. The overheads of OSGi may be a great solution for cell phones and for development teams within a single organization. That does not mean it is a great solution for us.

The "original context" or how OSGi was developed is inconsequential.  It is now
a leading java modularization standard and is widely used.

> (3) Before the flexibility of overrides, many participants in Kepler have found it impossible collaborate with each other. As a result, they tended to fork their own implementations of Kepler with whatever add-on functionality they wanted. If we had made the decision to go with OSGi, that would have been the end of overrides and would have lead to developers to tend to fork. The simple fact of the matter is, the negotiation costs of making every minor change you need in someone else's code is simply too high when you have a deadline to solve a particular scientific problem that is not primarily about Kepler, but used Kepler. 

You can do overrides in OSGi, I am amazed that you don't remember when I
discovered this!  See the bottom of this page
https://kepler-project.org/developers/teams/framework/design-docs/trade-studies/osgi-adoption/steps-for-kepler-conversion-to-osgi
that links to this page
http://wiki.eclipse.org/Steps_to_use_Fragments_to_patch_a_plug-in
and there are probably better sources about how to do this by now.

Negotiation costs can be handled many ways, certainly, overrides, also forking
(i.e. copying modules to new modules and modifying them), and even better!
actually talking to the original developers to help them improve their APIs
(imagine that!  people actually talking to each other).  This is a fact of
life, overrides are only a quick fix and spell disaster down the road when the
overridden code is changed, breaking the override.  We've already run into this
several times and it is a quick fix solution for a fix that could have been
handled easily and properly the first time.

> Another point is that OSGi was developed over 10-years has another implication which you probably would prefer not to talk about. It is old software. The thing about old software is that mistakes tend to be built into the software and are hard to get rid of. Take for example the issue of closures in Java. The reason that closures were not included in Java in the first place was because of time pressure related to doing the first release. But, it was found that callbacks in Java were simply too tedious. Having inadequate time to implement closures, the creators of Java settled on allowing anonymous class definitions with methods. This is an ugly solution, to say the least. But, it has remained in place. Only now, with the release of JDK 7 in September, is Java going to include support for closures. The thing about old software like OSGi or the JDK is that old mistakes or implementations built for a different context tend to persist. That OSGi is really really old at 10 years of age, is not necessarily an argument in its favor.

I agree, if there is a newer modularization system that has solved the problems
encountered in OSGi (every system has problems btw) then lets use that one. 
This is not about OSGi, it's about using a standard modularization system that
can work in other systems, and that allows us to use other components in our
system.

> (4) With the release of JDK 7, which is currently scheduled for September, there will be built-in Java language features that facilitate modules. Also, JDK 7 itself is being broken up using Project Jigsaw, a project which has language features being developed to support it and which will also be available to others who are interested in taking a similar approach to modularization. IF we were to use a third-party rather than custom approach to modules, it would be much more sensible to compare OSGi to the approach taken by Project Jigsaw. After all, there is a REASON that Sun decided to break up the JDK itself using a new approach rather than using OSGi. Apparently, they didn't want to be saddled with all of the complexities and old decisions of OSGi either. This should be a huge red flag telling you to look before you leap.

I have no problem with using Jigsaw, have you developed with it?  How does it
compare?  I have no experience with it. Do you think it would be easy to
convert to from the system that we have now?  Would it have been harder to
convert to from OSGi?  I am glad you are still looking at these other
technologies I hope that you keep up on them and let us know periodically what
is going on in that area.

> (5) Have you installed something using Eclipse lately? I have. I just installed a tool for the Google App Engine. And guess what. I had to restart Eclipse in order for it to work properly. It only partially worked before restarting. Hot deployment using OSGi apparently doesn't always work out so well after all. I m not blaming OSGi, but you would be a fool to think it magically solves our problems, really reduces complexity, or lessens the amount we need to understand with respect to such issues. As developers, we are going to have to research and understand in depth all the issues that make hot deployment successful or not successful and provide guidelines to module developers IF we even decide that this is that desirable of a feature, regardless.

Eclipse is not OSGi.  Do you think that when we introduce a bug into our Java
code that it is the fault of Java?  Your bad experience with poorly written
modules in Eclipse is not my concern, I have used many plugins for eclipse that
have worked just fine, and I have used several completely standalone
applications built on the OSGi standard, the equinox framework, and the Eclipse
Workbench bundles that work just fine.

> The bottom-line is this. The decision to not go with OSGi was carefully thought through, whether you agree with it or not. You had your chance to make your case. You failed to persuade others. That should be the end of it. Certainly, using Bugzilla right before the 2.0 release to air your grievances about decisions that have already been made and that you participated in is simply not optimal.

David, I am the only one that still tries to argue with you on these kind of
issues because I know that you have some training as a lawyer.  You were
trained to argue your case as either prosecutor or defense for one goal only,
and that is to win your case to the satisfaction of a judge or jury.  You have
been trained to gloss over truths that hurt your case and expose every possible
piece of evidence that helps your case.  And this is what you do, to the point
of making stuff up and saying things that just aren't true (like OSGi does not
support overrides, when it does).  I understand this and so I try my best not
to get upset with you.  But you should understand that in engineering, this is
a very detrimental approach.  The goal of a design engineer is to understand
all of his options as best as he possibly can in the time that he has.  He must
then make a choice from those options often without knowing all the details or
exactly what will happen in the future.  Often he must rely on his experience
and use his "gut feeling" to make a decision.  If he finds out later that his
choice is a good one, that solves his problem, then he continues down that
path.  If his choice does not solve the problem then he must accept that
another option should be followed.  There is no prosecution or defense here and
the only jury are the users of the system

You have worked extremely hard creating a very good build system and
streamlining the way in which developers build Kepler.  It has also allowed us
to come a long way in dividing up our codebase and managing our dependencies in
a top-down fashion. I really want to commend you on all that it has achieved. 
Like any system it has it's quirks but it works very well.  The issue that has
been poorly addressed is the runtime side of the module system.  This is what
we need to think about for the future.  How can we get our modules to run
smoothly at runtime such that a User can download a module and have it run
reliably with their Kepler instance?  Our module system kind of fakes that at
the moment by performing classpath magic on restart (as I understand it) which
leaves the system completely open to collisions (aka unintentional overrides). 
How does it address the 6 "Properties of a true module system" that Mirko
outlined in his article?  How can we get our system to perform robustly in that
kind of plug and play environment?  Is that something you want to develop from
scratch?

-- 
Configure bugmail: http://bugzilla.ecoinformatics.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.