I am a big user of ABS data, including for calculating educational participation and attainment rates. Recently I have been using Census longitudinal data, which links records from a 5% sample of the Australian Census between different Census dates.

Due to respondent inattention to questions, or mistakes by family members answering for others, I would expect some inconsistent answers between censuses. But inconsistency rates for education-related questions are alarmingly high.

For the highest year of school education the ABS-reported inconsistency rate between 2016 and 2021 was 6.8%. For highest non-school qualification the inconsistency rate was 8.9% – meaning that a lower highest education qualification was reported in 2021 than 2016.

Inconsistent answers for Australian citizens aged 25-34 years

To understand the implications of inconsistent qualifications for reporting policy-relevant attainment levels, I narrowed the results to Australian citizens age 25 to 34 years in 2021. The Census Longitudinal TableBuilder app flags 8.1% of this group as having a lower highest qualification in 2021 than in 2016.

The Census reports qualifications in a detailed way. Slight misremembers and variations in wording or coding of answers could drive up the inconsistency rate without necessarily changing results for analysis at broader qualification levels.

I classed a person as matched between 2016 and 2021 if the broad main tertiary qualification level for each year is in one of the following categories for each Census: doctorate, all types of masters, graduate certificate and diploma, bachelor, sub-bachelor diplomas and associate degrees, and Cert III/IV (ignoring Cert I/II).

This less strict definition of consistency reduces the inconsistency rate to 2.2%. Another 1.2% did not answer the highest qualification question in 2021 or inadequately described their qualifications.

Which qualifications have the highest inconsistency rate?

As the chart below shows, two qualification levels – doctorate and graduate certificate and diploma – have particularly high 2016-2021 inconsistency rates. In absolute terms the largest number of inconsistencies are for graduate certificates and diplomas, accounting for 31% of the 2016-2021 inconsistent responses, and certificate III/IV qualifications, accounting for 32% of inconsistent responses (this is the largest qualification category in 2016, which is why the low percentage error rate shown in the chart converts to a high number of people).

The following chart shows the lower qualifications cited in 2021 compared to 2016 levels.

Confusion about highest qualification

One possible source of error is that the Census asks about a person’s highest qualification, but the current AQF qualification hierarchy does not always provide an intuitive low to high rank.

For people who are unclear on qualification names and ranks the Census highest qualification question’s wording adds to rather than reduces confusion. Census respondents are prompted on the ‘associate diploma’, a relic of the pre-2001 qualifications framework, which the ABS does not include in Census results. Diplomas are mentioned twice but masters courses not at all. While the question asks about the ‘highest’ qualification its examples are not in order of lowest to highest or vice versa.

People completing the Census online in 2021 (79% in 2021) see the same wording, but if unsure can click on an information button which lists the current AQF order. I can’t see any statistics on how many people sought this information.

Diploma and certificate inconsistencies

Especially for holders of diplomas and certificates, with multiple within-category distinctions and versions both above and below a bachelor degree in the AQF hierarchy, qualification confusion is unsurprising.

About a quarter of 2021 inconsistent qualifications for people with a graduate certificate or diploma in 2016 were below-bachelor certificates III/IVs or diplomas in 2021. For people with a reported Cert III or IV in 2016, 95% of the 2021 inconsistent lower qualifications were in the Certificate II category. It’s easy to understand how graduate certificates and Certificates II, III or IV could blur in people’s minds.

Doctorate inconsistencies

For doctorate errors, a major likely cause is people whose AQF qualification level is bachelor or masters, such as medical professionals, but who commonly use the title ‘Dr.’ Some medical doctors also have PhDs, so it is hard to be definitive about the extent of this error. But I doubt, as the Census reports, that nearly a quarter of all medical practitioners have PhDs (there may also be errors in how people describe their occupation).

My hypothesis is that the lower qualifications reported in 2021 for 2016 doctorate holders are mostly corrections – these people are medical doctors whose highest AQF qualifications are masters or bachelor degrees.

Highest or most recent?

The medical practitioner chart above also shows, implausibly, that the highest qualification of some medical practitioners is below a bachelor degree. It’s possible that some people interpreted the question as asking about their most recent rather than highest qualification – people completing the Census are not necessarily reading the questions carefully.

The chart below is for 2022-23 and not confined to the 25-34 year age range, but for people holding both vocational and higher education qualifications it is common for a below-bachelor certificate or diploma course to be their most recent.

Unseen incorrect qualification upgrades

Nobody who reported a higher qualification in 2021 than 2016 is flagged as inconsistent, since it is possible that they completed another qualification in that five year period.

However the same confusions about doctorates, diplomas and certificates could lead Census respondents to report higher as well as lower incorrect qualification answers (as in the 2016 medical practitioner ‘doctorates’).

In the aggregate numbers incorrect upgrades and downgrades will, to some unknown extent, cancel each other out.

Effect on aggregated higher education attainment rates

Higher education attainment rates are often expressed as bachelor degree or above. Many of the inconsistent responses in 2021 for 2016 doctorate, masters and graduate certificate or diploma holder still report a bachelor or above highest qualification. At this level of aggregation, 1.5% of the people who reported this level of education in 2016 reported a below bachelor highest qualification in 2021. This gets us down to a far less concerning variation in numbers than the aggregate inconsistency rate.

Another way to test Census reliability is comparison with other data sources, such as the Education and Work survey. Education and Work is a sample survey, with the consequent risk that its respondents do not accurately represent the population. But Education and Work does not have a confusing highest qualification question. It offers online survey responses and telephone or face-to-face interviews. The latter methods allow an interviewer to clarify meaning if necessary. This contributes to lower rates of respondents not giving a substantive educational attainment answer (1.9%) in Education and Work than the Census (3.9%) (switching now to the full 2021 Census results, not just those with linked records to 2016).

If the people with no substantive answer on highest educational attainment are removed from each survey the attainment rates, on a broad qualification category basis, are very similar for 2021. The Census is slightly lower than Education and Work on bachelor degree or above and school or certificate I/II attainment and slightly higher on upper vocational qualifications (Cert III/IV and the vocational diplomas).

Conclusion

In future I will be more cautious about some uses of Census educational attainment data. Misclassified medical practitioners mean that a Census ‘doctorate’ cannot be assumed to be a PhD. Analysis of graduate certificates and diplomas similarly needs significant caution.

To improve Census accuracy the ABS should revise its Census question on highest qualification, to remove mention of now non-existent qualifications and list its examples in the same order as the AQF.

The ABS should also use its integrated administrative data system, which includes higher education completions as recorded by the Department of Education, to reality check reported increases in education levels. This can’t be 100% accurate as overseas and some local private provider qualifications are not in the administrative data, but large discrepancies would signal likely Census errors.

Despite scope for better results in later Censuses, current error rates on the key bachelor-degree qualification look low, and the overall bachelor or above attainment rates are similar to Education and Work. This means that the Census can be used for calculating overall higher education attainment rates.