The tyranny of complexity is a problem that has long been known to computer engineers. As long ago as the 1950s, the complexity of software systems was exceeding the limitations of humans to understand it. In the same decade psychologists were measuring human capability of dealing with complexity. George A. Miller's classic The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information famously showed that people could remember only 5 to 9 items of unstructured information. He gave the example of a "A man just beginning to learn radio-telegraphic code hears each dit and dah as a separate chunk. Soon he is able to organize these sounds into letters and then he can deal with the letters as chunks. Then the letters organize themselves as words, which are still larger chunks, and he begins to hear whole phrases." In other words, most of human effective short-term memory for low-information-content items was achieved mentally re-coding these items into a smaller number of high-information-content items, a process known as chunking.
This technique will sound familiar to software engineers who use abstraction and modular design to simplify the design of complex systems. Abstraction is not exactly the same thing as chunking but it addresses the same limits of human mental capacity. Software developers who work with high level languages like Java can work effectively without having to understanding the assembly code that Java translates to or the hardware actions that the assembly triggers. A high level language by itself does not bring the level of abstraction that developers require so they also use software libraries and frameworks .
However frameworks and libraries of ever increasing size don't completely solve the problem of complexity
they take ever increasing time to learn so
- the IT industry is filled with specialists who have learned one framework or another, and
- companies buy products and hope they work together
- take time to put together a group who can address a significant issue or construct a significant system, or
- if an existing group is used, then it will the solution will have a lot in common with previous solutions.
The software development industry has addressed this in various ways, some of which are
- Emphasizing workmanship over ephemeral results: ".. managing by results is, in effect, exactly the same as ... while driving your automobile, keeping your eye on the rear view mirror"
- Agile development breaks development down into short cycles with deliverables at each stage to allow efficient iteration to a working solution that meets the customer's needs.
- Software patterns acknowledge that there are effective methods that transcend computer languages and frameworks and attempt to harness them.
- The use of business analysts who define the problem to be solved and systems analysts who are responsible for researching, planning, coordinating and recommending software and system choices to meet an organization's business requirements.
- Outsourcing to companies with specialist expertise. e.g. IBM , HP , Fujitsu
- Virtualization, SaaS, cloud computing, etc
- SOA or Service-Oriented Architecture, is a yet higher level of abstraction which treats applications as loosely coupled inter-operable services whose functionality is defined along business processes. This includes orchestration which is the automated arrangement and management of complex computer systems, middleware, and services.
- See also SOA Sidekick Oriented Architecture for criticisms of SOA
- IT Governance frameworks
General Principles for Dealing with Complexity
People have been aware of the importance of pattern recognition in perception of the world since the time of Skinner's pigeons . In this recent article , Judith Polgar says "One of the biggest misconceptions about chess is it requires a lot of memorization. In reality, while some memorization is required, pattern recognition plays a crucial part in chess mastery." Most attempts at dealing with complexity that I am aware of come down to finding structure in the data being analyzed. The challenge is to come up with methods that find this structure quickly and reliably, like Skinner's pigeon and Polgar's chess players.
Effective developers fit the solution to the problem which requires
- understanding the problem being solved
- knowing how to solve similar problems
- an understanding of the underlying principles and technologies
- the ability to compare different methods
The following questions need to be answered
- what does my organization need to do over the next N years?
- how well can I predict that now?
- what is the most effective way I can spend my IT budget based on the information I have?
- If a major computer installation will take 3 years to complete for an organization whose needs for such a computer system change by 80% in 3 years then agility will be a key factor. However if the organization's needs do not change in that time or can be reliably predicted before the project starts then agility will probably not be critical.
- Another survey by Thomke and Reinerstein  showed that only 5% of product developing firms have complete product specifications before starting a design, and on the average only 58% of specifications are available before the design process begins. Barclay and Benson  cite that as many as 80% of new products fail, whereas about 25% of new industrial products and about 30-35% of consumer products fail to meet expectations.
- Platform middleware provides a number of facilities for declarative programming and middleware. ... ESBs also have a very similar form of deployment descriptor which enables loose coupling and metadata management on a per-service basis. ..business agility is impacted, because there is no business user interface for managing this metadata. ...
Products that Help Discover Structure in Information
These are all products that help people find patterns in their work. They are all complementary to the above well-known methods so I expect one of more of them to be widely adopted by IT organizations or framework builders.
- Mind Manager is an intuitive visual tool that allows users to create structure from unstructured information
- reQall generates reminder lists and has a cool iPhone interface
- Jott is like reQall and has voice recognition
- Evernote has some cool character recognition for converting scanned notes to data
- Leximancer drills into textual data ... and extracts the main concepts, themes and causal relationships to provide the information needed to make critical decisions. Some examples:
- Human Error in Maritime Operation says a large proportion of human error problems in the maritime domain can be grouped into a single category that can be labeled “loss of SA ”. ... Almost identical percentages were obtained between the manual coding and with the Leximancer analyses. ..Additionally, analysis of accident report forms can be successfully undertaken by using a tool such as the Leximancer.
- Technical description and evaluation
- Austhink Rationale
- NICTA managing complexity