The Tyranny of Complexity

The tyranny of complexity is a problem that has long been known to computer engineers. As long ago as the 1950s, the complexity of software systems was exceeding the limitations of humans to understand it. In the same decade psychologists were measuring human capability of dealing with complexity. George A. Miller's classic The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information famously showed that people could remember only 5 to 9 items of unstructured information. He gave the example of a  "A man just beginning to learn radio-telegraphic code hears each dit and dah as a separate chunk. Soon he is able to organize these sounds into letters and then he can deal with the letters as chunks. Then the letters organize themselves as words, which are still larger chunks, and he begins to hear whole phrases." In other words, most of human effective short-term memory for low-information-content items was achieved mentally re-coding these items into a smaller number of high-information-content items, a process known as chunking.

This technique will sound familiar to software engineers who use abstraction  and modular design to simplify the design of complex systems. Abstraction is not exactly the same thing as chunking but it addresses the same limits of human mental capacity. Software developers who work with high level languages like Java can work effectively without having to understanding the assembly code that Java translates to or the hardware actions that the assembly triggers. A high level language by itself does not bring the level of abstraction that developers require so they also use software libraries  and frameworks  .

However frameworks and libraries of ever increasing size don't completely solve the problem of complexity
they take ever increasing time to learn so

• the IT industry is filled with specialists who have learned one framework or another, and
• companies buy products and hope they work together 

This means that it will either

• take time to put together a group who can address a significant issue or construct a significant system, or
  
• if an existing group is used, then it will the solution will have a lot in common with previous solutions.
  

In 2008 software development parlance, the companies using these large frameworks are not as agile as they could be. Developing a new process might take so long that the the process may no longer fit the business needs it was meant to solve when it is complete.

The software development industry has addressed this in various ways, some of which are

• Emphasizing workmanship over ephemeral results: ".. managing by results is, in effect, exactly the same as ... while driving your automobile, keeping your eye on the rear view mirror"
• Agile development breaks development down into short cycles with deliverables at each stage to allow efficient iteration to a working solution that meets the customer's needs.
• Software patterns acknowledge that there are effective methods that transcend computer languages and frameworks and attempt to harness them.
• The use of business analysts who define the problem to be solved and systems analysts who are responsible for researching, planning, coordinating and recommending software and system choices to meet an organization's business requirements.
• Outsourcing to companies with specialist expertise. e.g. IBM , HP , Fujitsu 
• Virtualization, SaaS, cloud computing, etc
• SOA  or Service-Oriented Architecture,  is a yet higher level of abstraction which treats applications as loosely coupled inter-operable services whose functionality is defined along business processes. This includes orchestration  which is the automated arrangement and management of complex computer systems, middleware, and services.
• See also SOA Sidekick Oriented Architecture for criticisms of SOA
• IT Governance frameworks 

All of these methods work to some degree. Experienced software developers and managers have used these techniques or techniques like these long before the names were coined. The fact that these methods have recently been formalized and named means they are becoming available to a much broader audience such as developers working on end-customer IT installations.

General Principles for Dealing with Complexity
People have been aware of the importance of pattern recognition in perception of the world since the time of Skinner's pigeons . In this recent article , Judith Polgar says "One of the biggest misconceptions about chess is it requires a lot of memorization. In reality, while some memorization is required, pattern recognition plays a crucial part in chess mastery."  Most attempts at dealing with complexity that I am aware of come down to finding structure in the data being analyzed. The challenge is to come up with methods that find this structure quickly and reliably, like Skinner's pigeon and Polgar's chess players.

Effective developers fit the solution to the problem which requires

• understanding the problem being solved
  
• knowing how to solve similar problems
  
• an understanding of the underlying principles and technologies
  
• the ability to compare different methods

This sounds a lot like a seasoned developer would do and is the opposite of mechanically applying canned methods. It values understanding and flexibility over knowledge. However the best computer systems are complex so any practical approach has to combine sound technique with specific knowledge.

The following questions need to be answered

• what does my organization need to do over the next N years?
  
• how well can I predict that now?
  
• what is the most effective way I can spend my IT budget based on the information I have?
  

An example will illustrate how this affects the technique vs. knowledge decisions.

• If a major computer installation will take 3 years to complete for an organization whose needs for such a computer system change by 80% in 3 years then agility will be a key factor. However if the organization's needs do not change in that time or can be reliably predicted before the project starts then agility will probably not be critical.
  

Many studies including this one indicate that businesses are much less agile than they need to be. Here is an excerpt

• Another survey by Thomke and Reinerstein [10] showed that only 5% of product developing firms have complete product specifications before starting a design, and on the average only 58% of specifications are available before the design process begins. Barclay and Benson [11] cite that as many as 80% of new products fail, whereas about 25% of new industrial products and about 30-35% of consumer products fail to meet expectations.
  

Intentional SOA for Real-World SOA Builders describes how agility can be difficult to achieve in heterogeneous environments of complex frameworks

• Platform middleware provides a number of facilities for declarative programming and middleware. ... ESBs also have a very similar form of deployment descriptor which enables loose coupling and metadata management on a per-service basis. ..business agility is impacted, because there is no business user interface for managing this metadata. ...

In other words, a common low-level language (meta-data expressed in XML in this case) by itself is not enough to make two systems with different high-level semantics work together. It is revealing that the author thought he/she needed to spell this out. This implies that many or most businesses don't understand the need for common high-level semantics across systems that need to work together.

Products that Help Discover Structure in Information
These are all products that help people find patterns in their work. They are all complementary to the above well-known methods so I expect one of more of them to be widely adopted by IT organizations or framework builders.

• Mind Manager is an intuitive visual tool that allows users to create structure from unstructured information
  
• reQall  generates reminder lists and has a cool iPhone interface
  
• Jott is like reQall and has voice recognition
  
• Evernote  has some cool character recognition for converting scanned notes to data
  
• Leximancer drills into textual data ... and extracts the main concepts, themes and causal relationships to provide the information needed to make critical decisions. Some examples:
  
• Human Error in Maritime Operation  says a large proportion of human error problems in the maritime domain can be grouped into a single category that can be labeled “loss of SA ”. ...  Almost identical percentages were obtained between the manual coding and with the Leximancer analyses. ..Additionally, analysis of accident report forms can be successfully undertaken by using a tool such as the Leximancer.
• Technical description and evaluation
  
• Austhink  Rationale
  
• NICTA managing complexity 
  

I will attempt to quantify the ideas discussed here in a future post .