One for the Stats

Me mate Dave at work was much better than me at Pro Evo Soccer on the playstation, but you couldn’t tell that from looking at the after-match report. I’d get about the same number of on-target shots as him, comparable possession %, roughly the same number of corners, and basically on paper the game would look fairly even.

Anyone who actually watched a game, though, would see that he was streets ahead in skill, and pretty well dominated for 90 minutes. But not many people really watch the game.

Not as tenuous as some recent efforts to introduce an audit blog without turning the dial straight to ‘watching paint dry,’ this does, in some ways, point to another problem I have with the audit process – the count of audit issues as the primary indicator of the effectiveness of the control environment.

When the number of audit issues drops (compared to last year, say), clients assume that it reflects an improvement in their control environment, so well done, pats on the back all round, lets have some tea and scones.

WRONG! That’s not the way we do things. We sometimes split a single audit issues into several if we think it will be clearer that way, and sometimes when we find lots of related issues, we bunch them together into a single issue. And we do all of that without thinking about how it will look in some trivialised assessment of risk management later in the year.

Not that controversial or complex, but it really boils my pips. I saw this in an article a good while ago, and I thought I’d stick it in my blog and see if it contributes anything towards my stated mission of fixing all the world’s problems which are possible to fix by correcting audit client attitudes, behaviours, and miscomprehensions.

Truth #1

What gets measured is what gets done. Regardless of what you say or what you might consider to be common sense, people will do whatever they believe required to meet the metric.

Pasted from <http://www.itsmsolutions.com/newsletters/DITYvol5iss33.htm>

What its saying is that poorly designed metrics can drive people to behaviours that are inconsistent with or even obstructive to the goal you want to reach. And my comparison here is suggesting that a lazy reliance on raw number of audit issues as an indicator is also driving poor decisions and leading to unpleasant surprises in the foreseeable future.

Audit clients – please read between the lines. When you pressure auditors to reduce the number of issues in a report, that does not translate into an improvement in your control environment. When our stated conclusion (probably best expressed in the first draft we shared with you) says you have a problem, then you almost certainly have a problem.

Short post otherwise, so let me get back to my soccer story.

To get the ‘shots on target’ up, I pulled in a technique that I developed on my old Amiga back in ’92, the first 3D football game I’d seen. Goal! I think was its official title, but it was the third sequel to Kick Off, the main alternative to Sensible Soccer. So basically, when your full back or winger has the ball and runs over the half way line at either side of the pitch, you could shoot with full power and full after-touch to make the ball keep its height until the goal line and swing wildly across the face of the goal. I reckon I had about a 50% on-target rate with this, and about 10% scoring success. It pretty much never scores on Pro Evo (even against a really poor keeper), but still goes down on the stats. Very good way to get those shot-stats up without needing to navigate past the opponents defence or midfield (which I basically couldn’t do against Dave).

Possession stats can be enhanced by letting your keeper wait the maximum time with the ball in his hands every time he collects it. And then, rather than welly it down the pitch, just roll it to an unmarked defender and watch the seconds tick away – you can keep possession by short-passing it between your 4 defenders while the AI keeps the attackers to 1 or 2, so even against someone much better you can get that basic % possession up around the level you want it.

In all fairness, it was his playstation, so he knew the controls better than me. I’m sorry to say that’s the best and only excuse I have. And also it was the first version of the game where you could shove players off the ball rather than hold-X-to-tackle every time, which was pretty hard to master. All right, 2 excuses then.

Anyway, I’m aiming for a punchier top-10 type post next week, so if you’re free this time next Friday, tune in and see what’s going on in the murky world of audit and other stuff.