For instance, although return on investment is a useful metric, it is often not feasible to measure and track it for a piece of software. Thus, in practice, we settle for things that only paint part of the picture. It is important to acknowledge this. Often, enterprise IT loses sight of this truth. It deludes itself that the reported metrics constitute the big picture (From the school of “If you can’t measure it, it doesn’t exist.”).
What is worse, in the name of continuous improvement, these metrics are converted into targets. Teams now have an incentive to work towards local optima. No wonder large swathes of big enterprise IT are in a desperate mess. New causes of failure are discovered all the time. Heads roll and the new heads track a slightly different subset of metrics.
Automatically converting metrics to targets also has adverse psychological effects. Here is an example from day trading. What is more important? Making more winning trades than losing trades or making money? Obviously it is the second. But the moment you start tracking win-loss ratio it may become a target it itself. Emotional conditioning will also come into play and we will try to have less losing trades.
A software team can get severely constrained when a velocity target is imposed on it. At a certain threshold of constraints, team members lose a sense of empowerment. They play it completely safe and lose initiative. We experience this phenomenon sometimes when we deal with call centres. Government sponsored healthcare and education in western economies often exhibits similar characteristics. It is not just bureaucracy. In an effort to scale and centrally control efficient delivery of services, a raft of metrics is imposed on the practitioners. Performance of schools/hospitals is then tracked against these metrics (they now become targets). Teachers and doctors get frustrated, lose initiative and just play by the book much to the exasperation of parents and patients.
This effect is also known as Goodhart's law, after Professor Charles Goodhart who was Chief Adviser to the Bank of England. Restated more succinctly and more generally:
Trouble is, it is easy to scale management by numbers. It is very difficult to scale management by context. Perhaps it is worth asking if scaling is more important than delivering a quality experience. Besides, too much scale creates “too big to fail” monsters that have to bailed out with the money of the innocent in times of crisis. This is not just true of macroeconomics. It is true of macro IT as well.
A software team can get severely constrained when a velocity target is imposed on it. At a certain threshold of constraints, team members lose a sense of empowerment. They play it completely safe and lose initiative. We experience this phenomenon sometimes when we deal with call centres. Government sponsored healthcare and education in western economies often exhibits similar characteristics. It is not just bureaucracy. In an effort to scale and centrally control efficient delivery of services, a raft of metrics is imposed on the practitioners. Performance of schools/hospitals is then tracked against these metrics (they now become targets). Teachers and doctors get frustrated, lose initiative and just play by the book much to the exasperation of parents and patients.
This effect is also known as Goodhart's law, after Professor Charles Goodhart who was Chief Adviser to the Bank of England. Restated more succinctly and more generally:
When a measure becomes a target, it ceases to be a good measure.Measurements should not automatically become targets. When measurements indicate something is wrong, it calls for a conversation in context, not for a rating downgrade. Conversations in context may reveal that things are still ok in the larger scheme of things, the measurements only point to a local sub-optima.
Trouble is, it is easy to scale management by numbers. It is very difficult to scale management by context. Perhaps it is worth asking if scaling is more important than delivering a quality experience. Besides, too much scale creates “too big to fail” monsters that have to bailed out with the money of the innocent in times of crisis. This is not just true of macroeconomics. It is true of macro IT as well.
1 comment:
Hi Sriram,
The goal of software metrics is to provide a quantifiable methodology to assess the present state of software development process. Generally it is a valid tool for analysis. For example, team X has a productivity of 0.8 and organization wide productivity for the given programming language/platform is 0.95; why the difference? This question should lead to a root cause analysis to identify reasons for reduced productivity and solutions to achieve optimal productivity. The challenge appears when one has a single minded focus on one metric and while remaining blind to the impact on other metrics. For example if a sales man is given a target purely focused on sales and not on profitability, he/she will get big orders giving nice discounts or other freebies, but will it assist the organization if it is achieved at the cost of profitability? Actually this did occur in GE! Single minded focus on quality is also harmful; there is a price one pays for achieving quality. Therefore one needs to set goals considering the tradeoffs; typically it boils down to the quantity quality trade off. We would love not to have measurements but in practice these measurements ensure that we are objective in our assessment and can stand hostile scrutiny. The solution is to ensure that instead of a single measurement we set a group of measurements together as a goal. We do something similar is architecture design using the ATAM methodology. Deming also suggests so and a lot of things. Ref: http://findarticles.com/p/articles/mi_m1038/is_n5_v36/ai_14723299/
You are right taking measurements to extreme is detrimental, however management by context is not objective and subject to interpretation.On a small scale this is easy to manage, however in the context of large enterprise/government, a better approach would be to group goals together instead of looking at them individually.
Post a Comment
Note: Only a member of this blog may post a comment.