Category - Driving Tests

Driving Test Results Rigged?

UPDATED

I’ve updated this article, which was originally written 3 years ago (September 2010). The subject is going through one of its periodic revivals among the ignorati across various web forums.The sack

At the time I wrote the original, there was a story doing the rounds on these forums concerning someone who called himself Jim Kerr. He claimed he had been an examiner for 23 years, and he said:

I HAVE JUST BEEN SACKED BY THE D.S.A.. WHY? I HEAR YOU ASK.

BECAUSE, ACCORDING TO D.S.A.’S NEW “CHI” SYSTEM OF COLLECTING PASS RATES FOR EXAMINERS AT EACH TEST CENTRE, MY PASS RATE WAS HIGHER THAN THE OTHER EXAMINERS AT [named test centre] AND SO I MUST BE DOING IT WRONG.

There was much more, but that was the essence of his letter. Apparently, he’d put it into leaflet form and was handing it to people turning up for their tests outside the test centre he’d been sacked from. This act alone said a lot about the kind of person he must have been, and about the kind of person his employer had to put up with.

Anyway, when the story broke there was the deafening sound of several thousand colostomy bags all popping at the same time as ADIs logged on to the internet for their weekly surf and saw the story! You see, many in this industry sincerely believe that driving tests are rigged, and a story like this is guaranteed to get them hyperventilating. But let’s just do a reality check for a moment.

No company the size of the DSA (now known as DVSA) sacks someone on the spot for the sort of thing Kerr was claiming. They know bloody well that people like Kerr are “union” through and through, and they will only act if they’ve jumped through all the right hoops – or if they are sure of gross misconduct having been committed. What usually happens in a situation like the one Kerr is claiming is that retraining is given first of all. If that fails there will be further retraining. If that also fails then you’re into verbal warnings, written warnings, and final warnings. If there are still issues once all that has been done then dismissal is likely to follow. Companies who want to dismiss staff need a cast iron case and – except in the case of gross misconduct – a properly constructed paper trail.

Judging from the geographic location of the test centre in question, I’d be very surprised if Kerr hadn’t been in the union – and even if he wasn’t, if his claims came even close to being true he’d have been awarded a knighthood for wrongful dismissal at any tribunal! He certainly didn’t need to behave like a complete arsehole and picket the test centre to get justice.

In other words: there was probably something Kerr wasn’t telling us about his dismissal (see the original update to this story for confirmation).No cheating sign

But what about his claims? Is there any truth in them? Well, when I qualified my supervising examiner (also a test centre manager) told me this:

You’ll hear a lot of stories about quotas. They aren’t true.

What I do is monitor all my examiners, and if one of them has a significantly higher or lower variance than the rest I look into it. If we identify a problem then this is dealt with.

I am a scientist by training, and I understand maths and statistics. What my supervising examiner told me can therefore be summarised as follows (note that in this updated version I refer to a variance of 5% instead of the 10% that was being used at the time I wrote the original article):

    • over a period of time, with hundreds of tests conducted, you would expect every examiner to have roughly the same pass rate, give or take a few percent variance
    • the national average pass rate is around 45%, so over time you would expect the average pass rate for each individual examiner to be around the same figure, plus or minus a variance
    • if one examiner had a pass rate of say 40% and another had 50%, then it wouldn’t matter because they both fall within an acceptable band either side of the mean (i.e. ±5% variance)
    • if one examiner had a pass rate of, say, 20% or 60%, or if their variance was always skewed high or low, then this would point to a possible problem wit the way they were conducting tests
    • if the anomalous pass rate continued across several monitoring periods of, say, 3 months each, then a genuine issue with the examiner’s test performance would be confirmed

Most people don’t understand maths and statistics, and they cannot accept that a single examiner consistently having a grossly different pass rate to everyone else proves that there is a problem with that examiner. Kerr appears to be one of these ignorant people, believing his distorted understanding to be some sort of epiphany. As I say, I’ve known what happens since I qualified as an instructor and it is no big deal.

Kerr obviously couldn’t understand that his higher pass rate pointed to him doing his job differently to everyone else. The update to this story shows clearly that Kerr was being a little creative with the truth, and that he had been spoken to on several occasions. He simply refused to change, believing that he – as a minority of one – was right, and that all the other examiners out there were wrong.

There is one aspect of the system the DSA uses which has always worried me, though – and it isn’t the DSA’s fault. Not directly, anyway. Imagine this perfect world scenario:

  • over a typical monitoring period, an examiner has a pass rate of 60% – a variance of 15% above the average for his test centre
  • this is flagged to the examiner by his line manager, but over the next monitoring period nothing changes
  • the DSA uses its internal procedure to re-train the examiner, as he is clearly passing some candidates who ought to be failing (obviously, the situation would be the same if the examiner’s pass rate was 15% below the average for the test centre, and he was failing people who ought to be passing)
  • That last part is totally beyond the comprehension of many ADIs (and examiners like Kerr). However, it is absolutely correct and proper: if the pass rate for the test centre in question is 45%, and if everyone else is passing/failing at around the same rate (±5% variance), then someone who is consistently passing/failing at a much higher/lower rate must doing their job wrong. It’s a simple fact.Now imagine a real world scenario to see where the trouble potentially starts:
  • over a monitoring period, an examiner has a pass rate of 60% – a variance of 15% above the average
  • this is flagged to the examiner by his line manager
  • the examiner fears losing his job and decides to take matters into his own hands
  • the examiner keeps a tally of his passes and fails, and randomly fails a few people for minor (driver) faults if he needs to bring his variance down to acceptable levels (i.e. within the ±5% acceptable variance)

There is absolutely no way that this isn’t happening somewhere out there. However, the issue is not that it is happening – but rather that there is no way to stop it. No matter what system the DSA used it would still happen in one form or another between unscrupulous examiners eager to conceal their own incompetence. Ironically, if they did their jobs properly in the first place there’d be no problem for them to try and hide, and they’d all be within the ±5% variance. In spite of all that, it is vital to understand that those examiners fiddling their results are in a tiny minority, and this in no way supports claims of “quotas” or fixed results from ADIs and disgruntled learners. Of all the many hundreds of tests my own pupils have sat, there are perhaps just two that sound a little fishy – and since I do not sit in on most tests I am only going on what the pupil has told me. There is no way that the examiners in Nottingham are unique in their overall consistency, and people like Kerr are the exception rather than the rule.

But back to Kerr’s claims. As you can imagine, this Revelation in The Gospel According To Jim Kerr is Manna From Heaven for some ADIs. One one forum alone someone opines:

I guess the reason the DSA doesnt [sic] want the contents of the leaflet published is that, they dont [sic] want us to know about their ‘variances’ or the new “chi” sytem [sic] of making sure the DE’s have to work within a certain amount of passes/fails, and how the DE’s have to comply with them. Although many ADI’s have known about it for a long time.

It’s hard to get this many contradictions and misunderstandings into two sentences… unless you’re so excited your fingers and brain are not in proper sync!

A variance is something that happens – not something you apply. If the DSA has decided that a variance of greater than ±5% indicates a problem, then that is the system being used as a quality control check. As I have already explained, if the variance is consistently outside ±5% then there is a definite problem. They can call their system whatever they like: Harriet, Tarquin… or chi. The name doesn’t alter what it is.

Do driving examiners fail people deliberately?

The short answer is NO. They do not. They are not told to fail people as part of any quota, and this applies to the vast majority of examiners throughout the UK. However, there are corrupt people in all walks of life, and as I explained above, it is possible that some examiners – a tiny percentage – will attempt fiddle their pass rates in order to avoid being “told off” by their managers.

What is the 5% variance that examiners have to keep to?

There isn’t one. The ±5% variance is something that is measured – not applied. Examiners are not told to “keep” to anything, though they are expected to perform within certain limits as a quality control measure. If any examiner is misinterpreting this and deliberately fixing their results, then they are simply being dishonest.

Are examiners told to keep their pass rates within ±5%?

No. That ±5% is a range within which each examiner’s pass rate is expected to fall, and if they’re doing their job properly in the first place then their pass rate will fall within that range. If an examiner’s pass rate consistently falls outside of the ±5% range then the DSA has internal systems which it uses  to apply retraining because that examiner is not performing to an acceptable standard. Retraining is a minor issue, but less informed examiners might see it as a threat to their job security and therefore they might attempt to force the pass rate. This would be entirely the decision of the individual examiner. It is not a DSA policy.

Do quotas exist?

No. The idea of quotas is promulgated mainly by poorly educated ADIs and learners anxious to defend their poor performances on test.

What is the best day of the week to take your driving test?

There isn’t one as such. If you can drive, it doesn’t matter if you take your test on Monday morning or Friday afternoon. The idea that by Friday all the quota has been “used up” is nonsense. I’ve had lots of passes on Friday afternoons – just as I have on all other days.

Test Pass: 29/8/2013

TickWell done Sarah, who passed today first time with just 2 driver faults.After all that negativity you’ve been coming out with over the last few weeks, I have to say I told you so.

In fact, I had to force her to take her theory test as well as her practical – if I’d left it to her we’d still be taking lessons in December! But she’s been another one of those it’s been a pleasure to teach, and I am certain she’ll be a safe driver once she eventually makes her mind up over the car she wants and gets out there.

Test Pass: 27/8/2013

TickWell done Mitch, who passed today with 5 driver faults. This was his third attempt – he should have passed those other two times, but the pupils you have most faith in are often the ones who often end up doing something silly. Not this time, though. A well-deserved licence is on its way!

We had to do a circuitous route to the test centre. I heard on the news before setting out that a train had derailed in Carlton (today was the first full day the line was in operation after five weeks of closure for signal and rail upgrading). The Netherfield barriers were down and there were long delays. We managed to bypass all that – but reports are that the damaged line could take many days to repair, and also the new signals have been damaged.

More FOI Driving Test Rubbish

I’ve mentioned in various article recently about FOI requests, and how they are able to bring the average hack at local newspapers to orgasm in five seconds flat.

This one is quite amusing, because it is particularly inept in its attempts to interpret statistics. The headline trumpets that Blackburn is the hardest place in Lancashire to take your driving test. It then goes on to cite a pass rate of 47.7% compared to the average of 51.5% across the county (Lancashire).

That is statistical noise, and is meaningless. It’s even inside the 5% variance that the DSA uses to make sure its examiners are doing their jobs properly. But that doesn’t stop a driving instructor from saying:

Some other areas are definitely a lot easier, but I think it’s a very good thing to drive here. If you can drive in Blackburn you can drive anywhere.

Someone else who doesn’t realise that a 4% difference is meaningless, or that 47% and 51% do  not constitute “easy” and “hard”.

More Bad Pass Rates – But You Have To Read This To Believe It!

You often hear ADIs going on about only teaching people the bare minimum of skills required to pass the driving test instead of teaching them how to drive properly. The DSA’s strapline is “Safe Driving for Life”. So it beggars belief when you see a story like this.

Greater Manchester - with Failsworth and Hyde highlightedOn the surface, it’s just another FOI request blown out of proportion by some local hacks. But the really frightening part is the case studies they’ve dug up.

The article reports that Failsworth test centre has the lowest pass rate in Manchester, at 39.2% (bolstered by a really funny (not) reference to the name). It points out that Hyde test centre is 16% higher than this, at about 55%. As I mentioned recently, no one is ever prepared to refer to the population demographics in the areas they are comparing. Failsworth (marked as “A” on the map), being much closer to the centre of Manchester (and virtually part of Oldham), has a much higher proportion of non-UK national citizens. Hyde (marked as “B”) is out in the sticks, even though it is still part of Greater Manchester. Indeed, it is only about 6 miles from Glossop – which is so rural not everyone has electricity there yet!  The article also says that Buxton – 25 miles out into the countryside – has a pass rate of 61.4%. Obviously, higher pass rates are the story editors’ only focus – but it does illustrate my point.

But here’s the best part. They give several examples. Firstly, someone called Kate Emmott failed at Failsworth, and is now planning on taking her test at an “easier” test centre. She says that she got a “major” fault (marked as “serious” on the test sheet) for not driving in a bus lane. She says:

It was coming up to 10am and I was worried about it. I think I had a really strict tester to be honest.

If Manchester’s bus lanes are anything like the ones around here, their morning hours of operation are 7.30-9.30. Being “nearly” 10am is not the same as 9.30am, and failing to realise this is not the result of a “strict” examiner. It’s the result of being a bad driver.

Then there is the case of Emily Bleackley, who failed her test four times in Failsworth, and then passed “weeks later” in Hyde. The report says:

…her second fail last December was for ‘getting lost’, while her third attempt was scuppered when she slowed down to let a car pass. Her fourth attempt was down to bad ‘filtering’ with other traffic, she says.

So here’s someone else who cannot equate bad driving with failing your test. You don’t fail for “getting lost” – unless you get lost and then make bad mistakes. Slowing down to let people pass – probably on a busy road, since we’re talking almost the centre of Manchester – is dangerous and the sign of a frightened rabbit! And “bad filtering” almost certainly means not being aware of what others are doing, and changing lanes without proper safety checks. In spite of this, Ms Bleackley says:

…[my] instructor could not understand [my] fails either… She said she couldn’t believe I’d failed because my driving was completely up to standard. I was in tears after my tests at Failsworth.

I’m sorry, Emily, but you listed at least three good reasons why your fails were totally justified, and it is shocking that your instructor thought you were “up to standard” if you were so obviously unable to cope with other traffic. You weren’t up to standard – that’s why you failed, and for reasons which are absolutely clear. And it is worrying that you subsequently passed at a test centre where you’re shortcomings perhaps weren’t challenged. Are you going to drive exactly the way you did on those failed tests now you’ve got your licence? Like not driving in bus lanes, slowing to a crawl when you get scared, and weaving from lane to lane without checking first?

The DSA is quoted as saying:

Pass rates can be influenced by various factors. Some people may take more lessons and be better prepared for the test. Statistical factors can also play a part as the number of tests conducted at different test centres varies significantly.

However, every driving test is conducted to the same strict requirements. We train examiners to a high standard and closely monitor their work to ensure that all tests are assessed consistently across the country.

But nothing can hide the fact that some test centres – and their routes – are inevitably going to be easier, and this is always going to result in some people passing their tests when they have underlying issues with their driving. Although it isn’t the DSA’s fault, the two examples above provide clear evidence that two very substandard new drivers have been put on the roads by the system – and it is therefore the system which is at fault. Unfortunately, the system is too complex to be able to reliably identify every single variable involved, and it is left to people like Ms Bleackley and her driving instructor to open their mouths and provide the necessary pointers to where some of the faults with it might lie.

Another news story adds weight to this, and I’ll put an article about that together shortly.