Misleading Metrics, A Case Study
In the past, I worked for several large software companies, usually in the pre-sales consulting organization. Â One of them was particularly keen on metricsâ€¦ and striving to be â€œmetrics-drivenâ€. Â So, as you would expect, we had our objectives tied to Key Performance Indicators. Â The most important KeyÂ Performance Indicator was for the consultants to spend at least Â 240 hours of time in customer-facing situations (good idea actually!). The top people in each region were then awarded points which couldÂ be exchanged for pretty nice gifts from the company store (no, really they were Â nice). Â If you won, you could expect to receive @$2,500/year in addition to your normal pay. In general, this wasÂ a great concept which incentivizes the consultant to get out of his/her shell to help the people that pay your salary. In my opinion, the KPI was even set at the right amount of Â hours (about 50% of total available work time), because it was quite a stretch to spend that much time in customer activities, as we had plenty of paperwork, training and technicalÂ work to accomplish in that same period. Â Personally, I felt like I had a real advantage in this situation because of the Â relationships I had built in the region overÂ the years. Â So, to me this was pretty invigorating.
Then, reality came knockingâ€¦ hereâ€™s how it set in:
â€“ There were definitely 1 or 2 consultants who didnâ€™t have a problem fudging their number. Â We can categorize this as an integrity concern.
â€“ Every consultant had a different definition of â€œcustomer-facingâ€ time â€“ Did prep time count? Time thinking about the meeting? Travel time to and from the meeting?
â€“ Also, the manager was awarded points for total time in the field for all of his/her consultants, so he/she didnâ€™t really have a reason to argue against people whoÂ were optimistic about their time spent.
â€“ This information was entered into an sales force automation system (part of CRM) by the consultant. We were kind of biased â€“ right?
â€“ The manager had favorites â€“ Now, letâ€™s be honest. We all have favoritesâ€¦ Of course we do. You canâ€™t just turn off your human nature and stop liking/respectingÂ people that you actually like and respect. This favoritism ensured that tough questions werenâ€™t asked to certain individuals.
So what happened? I busted my hump to make it into the yellow (KPI could be Red, Green Yellowâ€¦ gotta love the traffic lights). Then, I was raked over the coals forÂ being yellow. To this day, in my opinion, I believe the 2 consultants who won the 1st quarterly contest were absolutely, positively not in front of customers as muchÂ as I was. This was a pretty small group, so I had a fairly good idea. Was this demoralizing? Definitely. Did I stop trying as hard? No, but I kept on bringing upÂ the topic on conference calls to the annoyance of everyone (including myself).
What are my takeaways from this situation given my chosen career in analytics? Â Well, Iâ€™ve always been attracted toÂ objective subjectsÂ (having direct reports and 7 children, makes me painfully aware of this reality every day), because people problems are much more complicated to solve.
â€“ Having a good metric and goal remain valuable, butÂ the peopleÂ really matter most.
â€“Â Beware of biasÂ in a metric at multiple levels in the organization.
â€“ Beware of carrots and sticks based on key metrics. Donâ€™t avoid them, but be certain that proper controls are in place to ensure compliance.Â GovernanceÂ can certainly help.
â€“ Ensure that the systems in place to capture the metric have a chance of getting the data without being subject to influence â€“Â External/ObjectiveÂ (this is covered inÂ our first blog in greater detail.) Northcraftâ€™s products give you an advantage in this area.
â€“Â TransparencyÂ â€“ This metric provided cover for the department and gave the appearance of transparency, but the truth was that 2 people were rewarded with incentivesÂ that shouldnâ€™t have been given to them.
How does this apply to IT?
Remember these concepts when calculating Availability SLAsâ€¦ and feel free to call us for help.
If youâ€™re interestedâ€¦
Weâ€™re been noticing through our own analytics that the most popular Northcraft content recently is:
#1: Coffee KPIs â€“Â https://www.northcraftanalytics.com/coffee-kpis-for-it/
#2: Service Management Success in Action Conference â€“Â https://www.northcraftanalytics.com/success-in-action/
#3: Metrics for Momâ€™s Minivan â€“Â https://www.northcraftanalytics.com/metrics-for-itsm-and-moms-mini-van/