Remove Inexcusable Complexity

Edward S. Lowry



THE PROBLEM:
Technical communication with people and machines is mired in inexcusable complexity. It needlessly impairs technical education, public safety, economic productivity, creativity, technical communication, and quality of life. Technical solutions that can broadly eliminate inexcusable complexity which results from poor language design have been available for decades but incentives to keep technology complicated have obscured them. The result is a metastable
situation where "small amounts of energy and resources precisely applied" can produce large advantageous change. This use of the trimtab principle could solve highly pressing problems and perhaps in a short time. Currently used methods of representation are unreasonable in much the way square wheels are unreasonable and they force producers of precise information to generate obfuscation.

The following sources of inexcusable complexity which now occur widely in language design can be greatly reduced.

1. Lack of structural expressiveness.
Insufficient "structural expressiveness" arises when the data structures of a computer language or any formal language are not rich enough to closely match the conceptual structures in some subject matter being modeled. When there is mismatch, extra data objects are needed. Languages like APL, Fortran, and SQL use only very constrained data structures and illustrate this deficiency.

2. Lack of functional expressiveness.
Languages like APL and SQL provide powerful operations so that a single expression can process several large aggregates of data. They may be described as functionally expressive. Most current computer languages such as Java, C++, Ada, etc (which have better structural expressiveness) severely lack functional expressiveness.

3. Technical learning load.
Failures to fix the above problems within a single language have resulted in a needless profusion of languages and computer interfaces. The computing sections of large bookstores present visitors with an uninviting mass of ephemeral detail.

4. Data structure mismatch.
This arises when information is worked on using multiple languages which have different data structures. Such mismatch occurs widely when a programming language and data base language are used together.

5. Use of data object structures which are not seriously designed to be easily arranged.
Arranging pieces of information is easier if they are designed to be easily arranged. They almost never are. Currently used pieces of information use traditional structures which lack any serious engineering design. This is analogous to a situation where architects and builders are oblivious to the value of using reasonably designed bricks.

6. Substituting informal expression for precision.
There is a traditional (but mostly unrecognized) failure to express precise information in precise ways. That now happens broadly wherever human intuition and effort can be used to compensate for poor language. Typical technical textbooks express information that could be expressed precisely using a mixture of precise mathematical formulas, natural language text, diagrams, and examples. The reader must distill precise information from the mixture and integrate the result in his/her mind.

7. Complexity defeats simplification efforts.
Complex systems tend to become incomprehensible and dangerous to change. That creates incentives to use shallow approaches to analyzing problems and that leads to further complexity and incomprehensibility.

Social aspects.
A fairly good understanding of the above technical problems and their solutions has been available for many years. However, many people have incentives to keep other people's lives complicated. For example, IBM declared 85% gross profit on $20B software revenue for 2007. Keeping software complicated makes it hard to switch vendors and that keeps the profits large. The result of such incentives has been 35 years of obstruction of simplification. The technical literature on serious simplification has been largely barren. See "Software Simplicity, and hence Safety - - Thwarted for Decades" on the web site (http://users.rcn.com/eslowry ). There are also incentives to avoid telling the truth about how inexcusable the complexity is. Nothing remotely like the truth has been publicly presented by anyone in a leadership position.

SOLUTION:
Organizing the list of 7 sources of inexcusable complexity as it is done above is part of the solution. There is a second trimtab effect. Fixing the first problems in the list clears away garbage complexity and that makes the remaining garbage more readily identifiable and easily removed.

Eliminating deficiencies of structural and functional expressiveness can be done by combining both capabilities in one language. Historically there has been a dichotomy between the two. Functional expressiveness depends on implicit iteration code which steps through the members of sets of objects. That depends on using a single dominant style of iteration. That goal has been defeated by the traditional way of achieving structural expressiveness which has added basic data structures and hence multiple styles of iteration. The dichotomy has never been fully circumvented in widely used technology. But there is no hard technical obstacle. By choosing a single pointer-like structure for basic data objects the dichotomy can be fully resolved. The PROSE language designed at IBM in the early 1970s (later revised, implemented, and then archived at DEC) demonstrated that structural and functional expressiveness could be combined.

That step also reduces the third, and fourth sources. The "generality" of a language can be taken as the class of applications for which it provides adequate simplicity of expression. Combining structural and functional expressiveness allows for a language of broad generality. The variations that are needed are largely at superficial design levels like syntax and vocabulary, so a single underlying design of data structures and basic functions for unrestricted general use in expressing precise information is technically practical. That view is heretical to many software leaders.

When one way of structuring data is adequate, the fourth source, data structure mismatch, is easily avoided.

The fifth source can be eliminated by imposing a series of constraints on the structure of basic data objects each of which averts a kind of needless complexity. Combining structural and functional expressiveness imposes severe constraints on what structures can be used. There are only a few realistic candidates. A single data object structure has been shown empirically to be highly effective. One of the candidates seems likely to provide a practical optimum for any sufficiently rich application. A rationale supporting hierarchically interconnected pointers as an enduring practical optimum is presented in "Toward Perfect Information Microstructures" on the web site. See Fig.1 and Fig.2. There are a few dozen engineering optimizations where reducing some deficiency in a single part to zero leads to a permanent optimum structure. Those include: round wheels, vertical pillars, tubular pipes, flat mirrors, etc. The optimization of data objects seems to belong to that class and will probably become similarly ubiquitous.

The sixth source, substituting informal expression for precision, can be reduced by using simpler, more readable precise information based on the above improvements. At present, students are massively taught how to arrange pieces of information without being given any clear idea of what is a reasonable structure for pieces of information. See "Formal language as a Medium for Technical Education" on the web site. The potential impact on teaching is not clear. Failure to provide students with a precisely expressed version of the precise information they are learning seems inexcusable.

The seventh source of complexity can be reduced by the preceding steps. One illustration of the power of this approach to clarifying technical knowledge is the discovery of a clear image of the classical electromagnetic field which lay hidden in Maxwell's equations for 90 years. See Fig.3 and "The Electromagnetic Field in Space-time" on the web site. It may express the essence of a physical theory better than any other image. The image resulted from translating Maxwell's theory into language for an infinite computer. That success in finding a larger simplification by first expressing more precise information simply has motivated most of the work since.

At least at a technical level, the solution is:
Comprehensive - it will obsolete much current software technology and representation of precise knowledge.
Anticipatory - it addresses the needs of an increasingly information intensive society.
Ecologically responsible - It creates no new burdens and will probably reduce some resource consumption.
Verifiable - comparisons of complexity can be very objective, especially after garbage complexity is significantly cleared away.
Replicable - almost anyone who is technically literate can develop and exploit the methods.
Achievable - progress in this direction seems inevitable - after the obstructionists are circumvented.

IMPLEMENTATION
Software implementation depends on substantial financing and circumventing a patent (#5664177). Expression of technical knowledge could draw on coordinated volunteer authorship in many technical areas.

FINANCING
Educational philanthropy may be a good prospect for funding. Persuading some prestigious technological organizations to tell the truth about how inexcusable complexity results from bad language would encourage investment.

NEXT STAGE
Progress seems to depend mainly on finding leadership with resources which truly values simplicity in technology and technical communication. Despite much searching no organization has been found which aspires to technology for expressing precise information simply that is as advanced as what IBM published over 30 years ago. Revision of a core definition for a language now called Shannon can proceed. Personal qualifications can be found in "Professional Background" on the web site.