Remedy unexplained 94% fail rate of over 100 ATPL candidates

Summary

• Sudden unexplained changes to ATPL Flight Planning subject in recent weeks has yielded an unprecedented failure rate of 94% among an estimated 100 to 200 candidates.

• It has caused hardship and expense to dedicated Pilots who sat the exam.

• The failure rate of 94% is unjustified by reference to section 9 of the Civil Aviation Act as it increases hardship for aviators but will not be likely to result in actual aviation safety improvements and is thus against the Australian public interest

• Relevant Examination Period: October until 20 November, 2012

• It is speculated (based on news of recent passes in the examination) that measures have since been taken to rectify the examinations since 20 November, 2012, however, that being so, it leaves those who sat the examinations up until then who have been unfairly treated without due acknowledgment nor remedy by CASA

• Your urgent intervention is sought to require the examiners at CASA to remedy the outcome of the unfair examinations to:

o standardise the examinations of those who sat the recent examinations so that the pass mark distribution is equivalent to if not better than the pass marks for examinations sat by candidates over previous years, and to reassess the examination results of the remainder; and
o investigate and review the specific examinations and marking methodology which were already notoriously hard to pass, so that the standard is restored for future candidates; and
o extend the three year limitation period for those whose limitation period is looming for the completion of their ATPL subject passes.

------------------------------------------------------

Relevant is the address by CASA Director of Aviation Safety (as he then was)
by John McCormick - 7 October 2009

INTRODUCTION

I am joining in this petition to protest against the conduct of unfair examinations set and marked by employees of CASA during October and up until 20 November, 2012.

RELEVANT EXAMINATION PERIOD: OCTOBER UNTIL 20 NOVEMBER, 2012

An earlier version of this petition was signed by candidates and their colleagues in Maroochydore, Queensland. I understand that of about 55 candidates who sat on 19 November, 2012, only 3 or 4 passed. Further news has come to light since, and some minor amendments are included in this petition.

The examinations were of candidates for the Airline Transport Pilot Licence in the subject of Flight Planning.

The usual pass rate for this examination is about 60%. This relatively low pass rate itself should itself be thrown into scrutiny and tempered by the principle of pragmatism stated by Mr McCormick above.

You will be aware that the passing of this subject has become more of an administrative rite of passage, notoriously hard, but a theoretical process that is never again called upon for ATPL licence holders to perform, neither pre-flight nor inflight.

Measuring the competence of a pilot based on lofty standards of accuracy for the performance of skills in a manner that in today’s aviation environment is not required to be done, is in contradiction of the principle espoused by Mr McCormick. Even a CASA set fail rate of 40% for the examination of these esoteric skills seems excessively high.

Yet, the newly issued examination pass rate (based on a sample of about 80 candidates) is believed during this period to have been about 6%.

The candidates have suffered serious detriment, and the intervention of the Minister and CASA Director is sought.

The remedy for which I plead is to give the benefit of the doubt to the candidates who sat this examination by standardising the examinations and granting a pass to an even higher proportion of more in line with that of ATPL Performance and Loading which I estimate to be 80%. This should be done having regard to the dedication of candidates being extremely high, and having regard to the fact that overall, candidates will still have had to demonstrate the competencies of all the other ATPL subject in order to be awarded the unfrozen ATPL.

Also, and although this will of necessity take some time to do thoroughly and fairly, the examination questions should independently vetted and validated, and the results of the remaining candidates not granted passes in the terms set out above, should be reassessed according to the validated answers.

HISTORICAL PASS RATE 60% AFPA – FLIGHT PLANNING

Historically, the pass rate of students in the subject has according to the CASA website been about 60% (based on period July to Dec 2010).

NEW EXAMINATION WRITTEN BY CASA EXAMINERS IN THREE DAYS (OR SHORT TIME)

I am told the CASA examination website was offline for three days or so that new examinations could be written following speculation that a “cheat sheet” was in circulation with respect to the foregoing examinations. CASA has undertaken a great deal of pressure to write new exams in such a short time, no doubt.

The CASA website indicates that in writing examinations a formal process is followed that includes “moderation”, “vetting” and “validation”. It is wondered whether these processes were followed in writing these examinations, because ordinarily examination questions take a considerable amount of time to write and to vet to ensure that they are accurate and appropriate.

PASS RATE 6% FOR NEW EXAMINATION

On 19 November, 2012, about 55 students sat the examination at Maroochydore, and only 3 of them passed (a fail rate of about 94%). I am advised that in Melbourne about 20 candidates in the past month sat the examination, and it had a fail rate of 100%.

In Melbourne, an equally appalling fail rate is evident to have taken place during the relevant period.

It is trite to say that this exceptionally high fail rate is a problem: but it either reflects poorly on the newly written CASA examination or it reflects poorly on the standards of those who sat the examinations.

NO NOTICE OF SYLLABUS CHANGED SINCE JULY 2000

There is a syllabus published on the website of CASA, entitled “The Australian Air Transport Licence (Aeroplane) Examination Information Book” and the edition is stated to be Version 2.2 – July 2000 (incorporating Reading List V1.0 – May 1998).

There is no indication or notification that the syllabus or standards have been changed since then.

To change the syllabus or expectations without notice to candidates and to the established Ground Theory Schools seems at the very least, unfair.

STATISTICAL PASS RATE EXPECTED TO BE THE SAME IF CALIBRE OF CANDIDATES THE SAME

Given that no notification had been given that the syllabus nor testing standards were to change, for a given sufficient sample of candidates, statistically, the pass rate ought to have been approximately the same.

I understand that in the relevant period students from different ground theory schools have sat the examination, and only three have passed.

I am aware that as of about 22 November, 2012, two candidates passed the examination, and am wondering whether measures this is a belated reaction by an individual or group of individuals at CASA to rectify the problem for the future without acknowledging a problem having arisen up until now.

It defies statistical feasibility that the standards of all these students in the relevant period were suddenly so drastically low compared to students who have passed the Flight Planning examinations in previous sittings.

CALIBRE OF THOSE WHO SAT

Included among the candidates was a range of not only experienced and skilled commercial pilots, but also respected military pilots.

WHETHER A NEW MEMBER OF CASA BELIEVES THE STANDARD OF CANDIDATES HAD BEEN TOO LOW AND NEEDED TO BE CHANGED

I understand, from the rumour mill, that a newly employed individual within CASA maintains that the standards of students has dropped and that a raising of the bar is justified. This letter does not rely on this assertion, but if it is true, it seems unfair that a change in standards has been imposed without adequate nor even any prior community and industry consultation and without altering the published CASA syllabus.

WHETHER OR NOT TOLERANCES TOO TIGHT, AND WHETHER OR NOT THE CASA EXAMINER’S ANSWERS WERE CORRECT AS AGAINST THE ANSWERS CHOSEN BY THE CANDIDATES

A majority of the candidates believe that they ought to have passed and that a sufficient amount of the answers that they indicated on the examination were correct. It appears that the examination may have been based on alternative but unpublished criteria, but whether or not this is so it is submitted that the examiner is possibly relying on tolerances that are too tight to be relevant to aviation safety, or that the examiner’s calculations were incorrect. On this point, further reference is made to the paragraph above headed “New Examination Written by CASA in three days (or short time)”.

Statistically speaking, such a deviation in the pass rate from previous groups of candidates is in the absence of a change in syllabus and methodology unexplained.

STATUTORY OBJECTIVES OF CASA DEFINED UNDER CIVIL AVIATION ACT

Common sense suggests that the objectives of CASA are to balance the needs of maintaining and seeking ways to improve safety in the Australian aviation industry, but to balance these objectives with the economic viability balancing these needs appropriately in the Australian public interest. This common sense position is reflected adequately in the Civil Aviation Act which states, in section 9, among other things:
• CASA has the responsibility of “encouraging a greater acceptance by the aviation industry of its obligation to maintain high standards of aviation safety”.
• At the same time, CASA must “promote full and effective consultation and communication with all interested parties”.
• Contextually, CASA must perform its functions “for the benefit of the Australian community and for export”.

RELATIONSHIP BETWEEN AN INCREASED SKILL THRESHOLD AND THE OBJECTIVES OF THE CIVIL AVIATION ACT

In accordance with the principles espoused by Mr John McCormick, it is always theoretically possible to increase safety and safety standards, however, there is a point where demands for increases in safety offer diminishing returns in safety, but adversely affect the economy and employment opportunities. Mr McCormick’s views on this gives us faith in presenting to him this petition that he will investigate the matter with CASA to ensure the examinations reflects this understanding.

Active airline and military pilots frequently advise that the syllabus of Flight Planning even as it was previously set, is a set of skills that are not performed by Airline Pilots. It is not contended for a minute that the Flight Planning course as previously examined is not useful nor necessary, but it is contended that the threshold as it had been set was already higher than anything that will be relevant to a contemporary pilot in the course of their future career. In other words, the syllabus as set more than equipped candidates for the ATPL to safely perform their duties and beyond.

DETRIMENT AND HARDSHIP TO THOSE WHO UNDERTOOK THE MOST RECENT EXAMINATIONS

The students who have attended the ground theory courses of reputable and experienced ground theory instructors have done so at substantial personal and financial expense. Their dedication is evident by the mere fact that they go to the effort and expense to enrol in and attend the classes.

Candidates who have attended such ground theory courses have:
• travelled from all over the country and some even from overseas
• many of them have taken time off work without pay
• many of them have paid for accommodation throughout the duration of the course
• most have attended all the course sessions
• most have studied furiously during their spare time outside class hours for the duration of the course

Some of those who have failed have three year examination time limits that are about to expire.

FAILURE TO FULFIL THE TRANSPARENCY CRITERIA OF THE CASA CHARTER

The CASA charter indicates that:

“We have a responsibility to inform people about their rights and responsibilities, including the right to expect high standards of service and behaviour from CASA officers.
We believe that people who are better informed and have a clearer understanding of legislative requirements are better able to comply with the rules and regulations in the maintenance of air safety. This in turn will enhance aviation safety levels in Australia.”

That being so, it would be expected that CASA would have communicated with leading industry educators to advise them of the standards (responsibilities) expected by CASA and the methods candidates are expected to follow in performing calculations in the examinations.

Despite this, no change was indicated about the upcoming examinations.

REMEDY SOUGHT – STANDARDISE THE EXAMINATION TO BRING ABOUT A PASS RATE COMMENSURATE WITH THE SUBJECT OF ATPL PERFORMANCE AND LOADING AND INDEPENDENTLY REVIEW ANSWERS TO EXAMINATION QUESTIONS

In summary, the recent CASA examinations have altered without notice the standard of knowledge or methodology that is required to be used in completing the Flight Planning examination. The failure to give notice is in breach of CASA’s own charter. Also the increased standard is beyond that which is likely to achieve an increase in safety but has the effect of causing personal and economic detriment to members of the Australian community including to members of the aviation industry.

These examination results impairs the reputation of CASA among members of the aviation industry undermining the need for a cooperative and constructive relationship.

To overcome this, decisive action is called for, to remedy without delay the loss suffered by the candidates who have sat these examinations by putting them in a position that they would have been in legitimately had the standard of the examinations been kept constant.

IN VIEW OF THIS AND THE UNFAIR DETRIMENT TO CANDIDATES, PLEASE REVIEW AND RESCORE THE EXAMINATION BY:
• STANDARDISING: immediately standardising the results in line with the pass rate of Performance and Loading, so that a fair proportion of the upper percentile who sat the examination are awarded passes for this subject; and
• REASSESSING: for the benefit of all remaining candidates, having the examination questions vetted and validated by a panel of active professional Australian Airline Pilots independent of CASA and issue passes to those who ought to have passed after the vetting and validation process is complete.

The fail rate anomaly potentially highlights that minor variations in methodology results in a divergence of calculation results that ought to broaden tolerances to more reasonably broad parameters. In the spirit of Mr McCormick’s words recalled at the commencement of this letter, the big picture issue should not be to maintain a slim fail rate, but rather to examine whether or not candidates have a relevant grasp of the principles and demonstrate sufficient competence to perform their functions and apply their knowledge safely as ATPL licence holders.

Each of these proposed remedy steps are fair, because those in the upper percentile have been assessed by one or more members of CASA who by CASA’s own reckoning are in the upper percentile and therefore that assessment should be relied upon for that purpose of standardisation. Also, the need for these candidates still to complete all other ATPL subjects will ensure as a cross-check they are at a high and safe enough standard of skill and knowledge to be awarded ATPLs.

The subsequent step is also fair because the remaining candidates in view of this extraordinary fail rate are entitled to expect that CASA subjects itself to independent validation and future candidates ought to be entitled to rely on the accuracy of the benchmarks against which they are being tested.

IT IS ALSO REQUESTED THAT FOR THE BENEFIT OF THOSE FOR WHOM THE THREE YEAR CUT-OFF TO COMPLETE PASSES IN ALL ATPL SUBJECTS IS LOOMING OR LAPSED, IT IS REQUESTED THAT AN EXTENSION OF, SAY, SIX MONTHS IS GRANTED.

The standard of most if not all of those who have attended and completed the syllabus of established ground theory courses is generally sufficiently high for the purposes of achieving the safety standards necessary for a person performing the duties within the Australian aviation industry of a holder of an ATPL Licence.

Time is of the essence in resolving this fairly, because in the meantime the many candidates who sat this examination are under stress awaiting a fair outcome, and are adversely affected in a variety of ways as a result of this most statistically excessive fail rate.

Summary of Actions Sought
1. Review and rescore the examination by:
a. STANDARDISING: immediately standardising the results so that the upper percentile who sat the examination are awarded passes for this subject in line with the pass rate for ATPL Performance and Loading; and
b. REASSESSING: for the benefit of all remaining candidates, having the examination questions vetted and validated by a panel of active professional Australian Airline Pilots independent of CASA and issue passes to those who ought to have passed after the vetting and validation process is complete
2. It is also requested that for the benefit of those for whom the three year cut-off to complete passes in all ATPL subjects is looming or lapsed, that an extension of, say, six months is granted.

Thanking you both for your consideration, and looking forward to a prompt response to the email addresses of the signatories of this petition.