UPDATED
I’ve updated this article, which was originally written 3 years ago (September 2010). The subject is going through one of its periodic revivals among the ignorati across various web forums.
At the time I wrote the original, there was a story doing the rounds on these forums concerning someone who called himself Jim Kerr. He claimed he had been an examiner for 23 years, and he said:
I HAVE JUST BEEN SACKED BY THE D.S.A.. WHY? I HEAR YOU ASK.
BECAUSE, ACCORDING TO D.S.A.’S NEW “CHI” SYSTEM OF COLLECTING PASS RATES FOR EXAMINERS AT EACH TEST CENTRE, MY PASS RATE WAS HIGHER THAN THE OTHER EXAMINERS AT [named test centre] AND SO I MUST BE DOING IT WRONG.
There was much more, but that was the essence of his letter. Apparently, he’d put it into leaflet form and was handing it to people turning up for their tests outside the test centre he’d been sacked from. This act alone said a lot about the kind of person he must have been, and about the kind of person his employer had to put up with.
Anyway, when the story broke there was the deafening sound of several thousand colostomy bags all popping at the same time as ADIs logged on to the internet for their weekly surf and saw the story! You see, many in this industry sincerely believe that driving tests are rigged, and a story like this is guaranteed to get them hyperventilating. But let’s just do a reality check for a moment.
No company the size of the DSA (now known as DVSA) sacks someone on the spot for the sort of thing Kerr was claiming. They know bloody well that people like Kerr are “union” through and through, and they will only act if they’ve jumped through all the right hoops – or if they are sure of gross misconduct having been committed. What usually happens in a situation like the one Kerr is claiming is that retraining is given first of all. If that fails there will be further retraining. If that also fails then you’re into verbal warnings, written warnings, and final warnings. If there are still issues once all that has been done then dismissal is likely to follow. Companies who want to dismiss staff need a cast iron case and – except in the case of gross misconduct – a properly constructed paper trail.
Judging from the geographic location of the test centre in question, I’d be very surprised if Kerr hadn’t been in the union – and even if he wasn’t, if his claims came even close to being true he’d have been awarded a knighthood for wrongful dismissal at any tribunal! He certainly didn’t need to behave like a complete arsehole and picket the test centre to get justice.
In other words: there was probably something Kerr wasn’t telling us about his dismissal (see the original update to this story for confirmation).
But what about his claims? Is there any truth in them? Well, when I qualified my supervising examiner (also a test centre manager) told me this:
You’ll hear a lot of stories about quotas. They aren’t true.
What I do is monitor all my examiners, and if one of them has a significantly higher or lower variance than the rest I look into it. If we identify a problem then this is dealt with.
I am a scientist by training, and I understand maths and statistics. What my supervising examiner told me can therefore be summarised as follows (note that in this updated version I refer to a variance of 5% instead of the 10% that was being used at the time I wrote the original article):
- over a period of time, with hundreds of tests conducted, you would expect every examiner to have roughly the same pass rate, give or take a few percent variance
- the national average pass rate is around 45%, so over time you would expect the average pass rate for each individual examiner to be around the same figure, plus or minus a variance
- if one examiner had a pass rate of say 40% and another had 45%, then it wouldn’t matter because they both fall within an acceptable band either side of the mean (i.e. ±5% variance)
- if one examiner had a pass rate of, say, 20% or 60%, or if their variance was always skewed high or low, then this would point to a possible problem with the way they were conducting tests
- if the anomalous pass rate continued across several monitoring periods of, say, 3 months each, then a genuine issue with the examiner’s test performance would be confirmed
Most people don’t understand maths and statistics, and they cannot accept that a single examiner consistently having a grossly different pass rate to everyone else proves that there is a problem with that examiner. Kerr appears to be one of these ignorant people, believing his distorted understanding to be some sort of epiphany. As I say, I’ve known what happens since I qualified as an instructor and it is no big deal.
Kerr obviously couldn’t understand that his higher pass rate pointed to him doing his job differently to everyone else. The update to this story shows clearly that Kerr was being a little creative with the truth, and that he had been spoken to on several occasions. He simply refused to change, believing that he – as a minority of one – was right, and that all the other examiners out there were wrong.
There is one aspect of the system the DSA uses which has always worried me, though – and it isn’t the DSA’s fault. Not directly, anyway. Imagine this perfect world scenario:
- over a typical monitoring period, an examiner has a pass rate of 60% – a variance of 15% above the average for his test centre
- this is flagged to the examiner by his line manager, but over the next monitoring period nothing changes
- the DSA uses its internal procedure to re-train the examiner, as he is clearly passing some candidates who ought to be failing (obviously, the situation would be the same if the examiner’s pass rate was 15% below the average for the test centre, and he was failing people who ought to be passing)
- That last part is totally beyond the comprehension of many ADIs (and examiners like Kerr). However, it is absolutely correct and proper: if the pass rate for the test centre in question is 45%, and if everyone else is passing/failing at around the same rate (±5% variance), then someone who is consistently passing/failing at a much higher/lower rate must doing their job wrong. It’s a simple fact.Now imagine a real world scenario to see where the trouble potentially starts:
- over a monitoring period, an examiner has a pass rate of 60% – a variance of 15% above the average
- this is flagged to the examiner by his line manager
- the examiner fears losing his job and decides to take matters into his own hands
- the examiner keeps a tally of his passes and fails, and randomly fails a few people for minor (driver) faults if he needs to bring his variance down to acceptable levels (i.e. within the ±5% acceptable variance)
There is absolutely no way that this isn’t happening somewhere out there. However, the issue is not that it is happening – but rather that there is no way to stop it. No matter what system the DSA used it would still happen in one form or another between unscrupulous examiners eager to conceal their own incompetence. Ironically, if they did their jobs properly in the first place there’d be no problem for them to try and hide, and they’d all be within the ±5% variance. In spite of all that, it is vital to understand that those examiners fiddling their results are in a tiny minority, and this in no way supports claims of “quotas” or fixed results from ADIs and disgruntled learners. Of all the many hundreds of tests my own pupils have sat, there are perhaps just two that sound a little fishy – and since I do not sit in on most tests I am only going on what the pupil has told me. There is no way that the examiners in Nottingham are unique in their overall consistency, and people like Kerr are the exception rather than the rule.
But back to Kerr’s claims. As you can imagine, this Revelation in The Gospel According To Jim Kerr is Manna From Heaven for some ADIs. One one forum alone someone opines:
I guess the reason the DSA doesnt [sic] want the contents of the leaflet published is that, they dont [sic] want us to know about their ‘variances’ or the new “chi” sytem [sic] of making sure the DE’s have to work within a certain amount of passes/fails, and how the DE’s have to comply with them. Although many ADI’s have known about it for a long time.
It’s hard to get this many contradictions and misunderstandings into two sentences… unless you’re so excited your fingers and brain are not in proper sync!
A variance is something that happens – not something you apply. If the DSA has decided that a variance of greater than ±5% indicates a problem, then that is the system being used as a quality control check. As I have already explained, if the variance is consistently outside ±5% then there is a definite problem. They can call their system whatever they like: Harriet, Tarquin… or chi. The name doesn’t alter what it is.
Do driving examiners fail people deliberately?
The short answer is NO. They do not. They are not told to fail people as part of any quota, and this applies to the vast majority of examiners throughout the UK. However, there are corrupt people in all walks of life, and as I explained above, it is possible that some examiners – a tiny percentage – will attempt fiddle their pass rates in order to avoid being “told off” by their managers.
What is the 5% variance that examiners have to keep to?
There isn’t one. The ±5% variance is something that is measured – not applied. Examiners are not told to “keep” to anything, though they are expected to perform within certain limits as a quality control measure. If any examiner is misinterpreting this and deliberately fixing their results, then they are simply being dishonest.
Are examiners told to keep their pass rates within ±5%?
No. That ±5% is a range within which each examiner’s pass rate is expected to fall, and if they’re doing their job properly in the first place then their pass rate will fall within that range. If an examiner’s pass rate consistently falls outside of the ±5% range then the DSA has internal systems which it uses to apply retraining because that examiner is not performing to an acceptable standard. Retraining is a minor issue, but less informed examiners might see it as a threat to their job security and therefore they might attempt to force the pass rate. This would be entirely the decision of the individual examiner. It is not a DSA policy.
Do quotas exist?
No. The idea of quotas is promulgated mainly by poorly educated ADIs and learners anxious to defend their poor performances on test.
What is the best day of the week to take your driving test?
There isn’t one as such. If you can drive, it doesn’t matter if you take your test on Monday morning or Friday afternoon. The idea that by Friday all the quota has been “used up” is nonsense. I’ve had lots of passes on Friday afternoons – just as I have on all other days.