This topic was triggered by Baris Kocabasoglu as he shared his experience on the use of metrics in configuration management during the December edition of the CM2 Congress. So for my first article of the new year, I’m going to explore the usage of metrics in Configuration Management.
Many configuration management professionals will recognize that senior leadership with the best intentions do not always choose the right metrics to measure the performance of a process or activity. Choosing the right metric is critical for enabling continual improvements because the wrong metric can steer you in the wrong direction. We all know that:
“The road to hell is paved with good intentions and the wrong metrics!”
Metrics: Cycle time
A metric that is often used, is the cycle time of changes with the intent to reduce the cycle time. While cycle time can be a good metric when you have an identical repetitive task, when you apply it to changes it can trigger the wrong behavior. A change is never the same, whereas assembling part A and part B to become Assembly C, is basically the same task over and over. You can try to improve the efficiency of that task by improving tools, improving work instructions, or training your employees. But the result of that task is always the same, namely assembly C. In such case cycle time is a good metric.
The cycle time of a change depends heavily on product complexity, the size, complexity and priority of the change. So, if people start scoping changes to be smaller you have more changes, with a reduced cycle time. This results in getting results slower as people need to manage the dependency between these changes which increases the overhead.
Note that cycle time is typically defined as the time to complete a single task (including all the wait times). A single change consists of multiple tasks, so measuring cycle time of a change basically means you are measuring its lead time. Not all single tasks in a change process are always the same, for instance the impact analysis. Does that mean you should not use a cycle time metric in your change process? No, but you should identify the true repetitive parts of the process that have the same outcome every time. So, for instance, you could still measure cycle time for a change request between ready for approval and dispositioning (reject, approve or return for clarification). The end result is in most cases a dispositioned change request. Note that you might want to measure the first-time right change disposition as well. It can happen that the change board sends the change request back for further clarification. That means a corrective action is needed to improve the change request definition, before it can be dispositioned.
Better use other metrics
But it is better to have metrics on planned versus actual with respect to the scope, cost (includes business case), and required resources (this includes time and people) of a change. This is an indication of the maturity of the teams and organization.
A way to tell something about the overall effectiveness of your configuration management practices, is the number of non-conformances that are reported.
You can use different perspectives here:
– # non-conformances / # products
– # non-conformances per product
– # non-conformances / # FTE
You can add filters like severity of the non-conformance to see how the results are distributed.
Due to the nature of these metrics, there is a risk that people would want to reduce the registration of non-conformances. However, when you do not use these metrics as a stick, but just to create awareness, you can mitigate this. You are not measuring the performance of a specific task but the result of an entire delivery system. It is not about punishing, it is about reflecting and improving.
Cesar Duenas made a very good point about normalizing metrics for non-similar products:
“For change management, I suggest the quantity of configuration items (CI) in your product or end item baseline (or a similar size metric) can be used to identify similar products and normalize metrics for non-similar products e.g. changes, errors or change effort per CI.”
Source: LinkedIn – How do you select key performance indicators?
Early design vs mature design
In relation to Cesar Duenas point, it is also important to understand in what lifecycle phase the product is in. In the early stages of new product introduction, you probably want to measure different things. A lot of early design work is highly iterative so measuring first time right is not really the right way to go. But measuring the trend with respect to the non-conformances gives you an indication whether or not the design is maturing or not. And when the time is right to bring the product under full configuration control.
But ones the product is under full configuration control the first-time right metric becomes important. This metric measures the number of change requests that fix a problem instead of introducing something new or an improvement (providing more value to your customers). This essentially will give us an indication of how well we implement our changes. The closer to zero the better you are in implementing changes.
Find the bottleneck
Something many configuration management professionals hear regularly is:
“The change process is too slow and cumbersome”.
But often it is not the change process that is slow and cumbersome, it is the engineering process or tooling that does not allow the engineer to be efficient. Or the reviewers of the technical documentation are taking too long or are on vacation and did not assign a delegate. Note: review of technical documentation changes is not part of the change process, this is part of the engineering process. The change process is only managing the overall orchestration of the change, the impact analysis, disposition of the change, the planning and release. During release we might check if the right people reviewed as per the requirements set in the engineering process, but we do not review the content. It is therefore important that the metrics we choose support us in identifying the true bottlenecks. Whether it is the change process or something else, knowing the true bottleneck allows us to improve.
What do you think?
Don’t forget to subscribe to this newsletter and follow me!
Header Photo by Maxim Berg on Unsplash