Complexity theory describes the domain between deterministic order and randomness which is complex.
It is commonly categorized into five domains:
- Simple domain
- Complicated domain
- Complex domain
- Chaotic domain
- Disorder (actually not a domain but rather something that does not fit in a single domain).
Disorder is where no knowledge of causality exists.
The simple domain could be simply described as 'cause-effect'. There is a simple and clear relationship between a trigger condition or input and the resultant output. A simple domain problem (or challenge) would be able to be solved in a particular way, commonly this is called a Best Practice, which will lead to the best possible result for the particular problem space. For example, computing the result of multipying two numbers, calculating the price for car insurance, or the tax due, are simple domain problems.
The complicated domain can be regarded as the problem space in which the relationship between cause and effect requires analysis or some other form of investigation and/or expert knowledge. It has several trigger conditions or inputs and/or several outputs. A complicated domain problem requires several processes to be performed in order to get a result. For example underwriting an insurance policy for industrial or commercial use might require claim history research, risk survey, and risk engineering, and the policy underwriting itself to arrive at a result. There are several ways to achieve a good result, no one best way. This is commonly called Good Practice. In some cases (but not all!) a complicated sub-domain problem space may be able to be reduced to a set of simple domain problem spaces, each with a Best Practice.
Chaotic systems can be regarded as a subset of complex systems distinguished precisely by an absence of historical dependence (not predictable). Chaotic behavior is the result of a relatively small number of non-linear interactions. For example, the weather is a classic chaotic sub-domain problem space. We know that certain patterns of high pressure and low pressure lead to particular temperature, wind and precipitation probabilities. Automobile traffic is also a chaotic sub-domain problem space, anyone who drives the same commute daily is subject to unpredictable variations even though the overall problem space, e.g. highway, is the same. Chaos mathematics provides a way to understand chaotic sub-domain problem spaces in general but not in specific detail.
The complex domain, can be regarded as the problem space in which the relationship between cause and effect can only be perceived in retrospect, but not in advance. For example in creating software, when you have discussions, or start writing tests, and you perceive that the requirements begin changing because of what you discover as a result, that’s a good indicator that the software problem space is complex. You can look back in retrospect at the result and understand that it’s better than you originally expected, but you can’t plan it, nor can you define what “better” will look like and try to reach it. It emerges as you work and your problem space knowledge increases.
When the problem space is particularly subject to emergence, this is an example of a 'Rugged landscape'. Innovation occurs usually in this landscape. When the problem space leads to a breakdown of predictability, this is referred to as the 'edge of chaos'. If you are trying to be creative (rather than just innovative) then it is likely you will be 'skating on the edge of chaos'. This is a a good way to perceive creativity as it implies a breakdown in known rules, restrictions and controls. Companies like Google are always seeking the 'edge of chaos'. They know when they reach this because the changes created are causing some form of chaos in the workplace. There was a great quote by a senior VP, she stated that if she was coming to work and something was not causing chaos, then they were controlling the employees too much.
Complexity science on its own is insufficient to handle a situation or problem space. It is great for recognizing whether a situation or problem space or challenge is simple, complicated, complex, or chaotic. But what do you do when have classified the domain?
If it is simple, look for practices that worked in the past. Is there a practice that seems to always lead to the best outcome? This is a likely candidate best practice. But don't assume it really is, it may just be the best practice known or tried up till now. Kaizen what you think is the best practice to see if it can be improved.
If it is complicated, then look for root causes, not just one cause, and look at practices that worked in the past. Continual improvement (from quality management, and The Toyota Way) usually will help you identify good practices, there will be several, that can be used in a complicated domain. There is NO best practice, rather there are several good practices. Ignore the Lean people who get it wrong when they propose best practice(s). Instead focus on processes that help drive the best possible result.
If it is chaotic, you need to sense the prevailing conditions or factors that influence (not direct) the system's behavior. For example, weather is a chaotic domain situation and it is well known that the pattern of high pressure and low pressure cells influences the overall weather pattern. In road traffic, it is the number of vehicles and the impediments (traffic lights, crossing traffic, and weather) that influence the overall traffic flow. Look for approaches that help shape a response to emergent conditions, in other words sense the current conditions and respond.
If it is complex , you need to look at what parts of the system's behavior are simple or complicated, compartmentalize these and handle them as above. Then look at what is chaotic and the remaining part is complex. Again sense the current conditions and respond to emergent behavior in an active manner. The state of the art for handling complex domain situations is to gather people with all expected or required knowledge to handle the situation or needs. Provide them with an environment where they can make decisions and continually adjust as the system responds to their actions and the conditions generating the complexity. A typical complex situation is a hostage situation. Some parts of it can be handled via good practices, some parts require skilled people. Another complex situation is when enterprises want to be inventive or highly innovative and break the current rules. In management, holacracy is one such approach that senses and responds. Symphonic collaboration is an approach that looks at how to bring people together to solve problems or overcome challenges that no single team or person can solve on their own.
How can you learn to handle complexity. The best way is to learn an Agile approach. Most Agile methods define how to create teams to handle situations and respond to change. The investment in training, and even more importantly in coaching is repaid many times over. I have worked with teams that have achieved 1300% productivity improvement over the way they worked previously.