A Simple Way to Improve the Scoring of Aged Care Quality of Life and Care Experience

Aged care services across Australia are required to measure how satisfied their residents are with their care and quality of life. However, there’s a problem with how these satisfaction scores are currently calculated when residents can’t answer certain questions. In the National Aged Care Mandatory Quality Indicator Program (NACMQIP), aged care services are required to administer a survey to Aged Care Consumers (ACC) measuring Quality of life (QOL-ACC) and Quality of Care Experience (QCE-ACC). The QOL-ACC and QCE-ACC consist of six questions each shown in Table 1. All questions are multiple choice with the options always, mostly, sometimes, rarely, and never, which are associated with scores 4, 3, 2, 1, and 0, respectively. However, two of the questions also have a “not applicable” option. The responses scores for QOL-ACC and QCE-ACC are summed and then categorised based on Table 2.¹ When the “not applicable” option is selected the NACMQIP manual states that we treat the response as a missing value. Their formula for imputing this missing value is to take the average of the five non-missing questions. In this article I show that we should be using separate equations for QOL-ACC and QCE-ACC.

Table 1: The questions from QOL/QCE-ACC, at the time of writing. All questions have the options always, mostly, sometimes, rarely, and never, but some questions also have a “not applicable” option.

	Question	Can be N/A
QCE-ACC
1	I am treated with respect and dignity:	no
2	I am supported to make my own decisions about the care and services I receive:	no
3	I receive care and support from aged care staff who have the appropriate skills and training:	no
4	I receive services and supports for daily living that are important for my health and wellbeing:	no
5	I am supported to maintain my social relationships and connections with the community:	no
6	I am comfortable lodging complaints with confidence that the appropriate action will be taken:	yes
QOL-ACC
1	I am able to get around as much as I want to (with the use of mobility aids, e.g. wheelchair, walker, stick if you use them, or other people who help you):	no
2	When I experience pain, it is well managed:	yes
3	I am generally happy:	no
4	I have as much independence as I want:	no
5	I have good social relationships with family and friends:	no
6	I have leisure activities/ hobbies I enjoy:	no

Table 2: The final category is determined from looking up each score in this table.

Category	Score Range
Excellent	22-24
Good	19-21
Moderate	14-18
Poor	8-13
Very poor	0-7

The current approach in the NACMQIP manual is presented as a lookup table similar to Table 2. There is an equivalent table for the “Pain is well managed” question. To make it easier to describe the implied relationship I will define some notation. Let \(x\) be the sum of the non-missing questions scores in QOL-ACC and QCE-ACC. Let \(y\) denote the question that can have missing responses. Then the NACMQIP manual’s current approach can be written as \(y \approx \frac{x}{5}=0.2x\).²

Table 3: QCE-ACC score with Not Applicable marked for “Lodge Complaints” Rescaled to 0-24. I have added the second column for clarity. The QOL-ACC lookup table is equivalent.

Summative Score Calculated	Estimate of “Lodge Complaints”	Rescaled Score
0	0.0	0.0
1	0.2	1.2
2	0.4	2.4
3	0.6	3.6
4	0.8	4.8
5	1.0	6.0
6	1.2	7.2
7	1.4	8.4
8	1.6	9.6
9	1.8	10.8
10	2.0	12.0
11	2.2	13.2
12	2.4	14.4
13	2.6	15.6
14	2.8	16.8
15	3.0	18.0
16	3.2	19.2
17	3.4	20.4
18	3.6	21.6
19	3.8	22.8
20	4.0	24.0

We can consider how well this approach works for the data that isn’t missing to get an idea of how good it is at estimating the missing value.³ Using NACMQIP data submitted by MOA Benchmarking on behalf of members we find the empirical relationship shown in Figure 1.⁴ In each case, there is a clear relationship between the question of interest and the other questions. The relationship appears to be approximately linear, especially when taking into account the greater uncertainty for lower values of \(x\).

Figure 1: The average score in the question that can be marked as “N/A” grouped by the sum of the other scores along with a 95% confidence interval. Only sum of scores with at least 50 data points were included.

Since it appears the relationship is approximately linear I propose using an equation of the form \(\mathrm{score} = \mathrm{intercept} + \mathrm{slope}\times\text{(sum of other questions)}\), or my existing notation, \(y=\mathrm{intercept}+\mathrm{(slope)}x\). Note that the current approach in the NACMQIP manual is of this form (with slope=0.2,intercept=0), but it is not the best solution for both questions. Below is an interactive graph where you can change the intercept and slope so that the line fits the data better. The current approach happens to work quite well for QCE-ACC, but not QOL-ACC. We need different equations for each question to fit the data properly.

Note

Reveal what I consider the “correct” answer by selecting the respective linear models from the preset drop down.

pain = await d3.csv("pain-empirical.csv", d => ({
  x: +d.x,
  y: +d.y
}))
feedback = await d3.csv("feedback-empirical.csv", d => ({
  x: +d.x,
  y: +d.y
}))
presets = [
  {label: "User defined", value: {slope: 0.2, intercept: 0}},
  {label: "NACMQIP Manual", value: {slope: 0.2, intercept: 0}},
  {label: 'Linear model for "Pain is well managed" (QOL-ACC)', value: {slope: 0.1, intercept: 1.87}},
  {label: 'Linear model for "I am comfortable lodging complaints" (QCE-ACC)', value: {slope: 0.19, intercept: 0.1}}
]
viewof preset = Inputs.select(presets, {
  label: "Preset",
  format: x => x.label,
  value: presets[1]
})
viewof slope = Inputs.range([0, 1], {
  value: preset.value.slope,
  step: 0.01,
  label: "Slope"
})
viewof intercept = Inputs.range([0, 4], {
  value: preset.value.intercept,
  step: 0.01,
  label: "Intercept"
})
html`The currently plotted equation is: y = ${intercept} + ${slope}x`

{
  if (slope!==preset.value.slope || intercept!==preset.value.intercept) {
    viewof preset.value = presets[0];
  }
}

Plot.plot({
  marks: [
    Plot.dot(pain, {x: "x", y: "y", fill: "red", r: 4}),
    Plot.dot(feedback, {x: "x", y: "y", fill: "blue", r: 4}),
    Plot.line([{x: 0, y: intercept}, {x: 20, y: slope * 20 + intercept}], {
      x: "x", 
      y: "y", 
      stroke: "black", 
      strokeWidth: 2
    })
  ],
  color: {
    legend: true,
    domain: ["Pain is well managed (QOL-ACC)", "I am comfortable lodging complaints (QCE-ACC)"],
    range: ["red", "blue"]
  },
  x: {label: "Sum of Other Questions", domain: [0, 20]},
  y: {label: "Average Score", domain: [0, 4.5]}
})

Figure 2: The average score of “Pain is well managed” and “I am comfortable lodging complaints”, grouped by the sum of the other scores and a line that can be modified interactively.

Unlike in the above exercise, statisticians aren’t visually finding a line that just looks right. Instead we select the optimal slope and intercept such that prediction error is minimised on the underlying data.⁵ The results of this is shown in Table 4. The linear models are visualised in Figure 3 along with the current approach. The visual and numeric output suggest that modelling the questions separately is appropriate. Furthermore, I found that the linear model is only 0.10% and 0.71% worse than a fully saturated model for QCE-ACC and QOL-ACC respectively, whereas the current approach is 1.67% and 18.27% worse.⁶ This means that \(y=1.87+0.1x\) (QOL-ACC) and \(y=0.1+0.19x\) (QCE-ACC) can’t be improved on much.⁷ There may be some bias since these formulas were determined based on MOA Benchmarking member’s data only, but the general conclusion that separate modelling of questions is appropriate would not change.

Table 4: The intercept and slope from a linear model fitted to the underlying data for the two questions of interest, with the sum of scores in other questions as a predictor.

Model for	Intercept	Slope
Pain is well managed (QOL-ACC)	1.87	0.10
I am comfortable lodging complaints (QCE-ACC)	0.10	0.19

Figure 3: The proposed linear models, the current NACMQIP Manual implementation, and the average score of “Pain is well managed” (blue) and “I am comfortable lodging complaints” (red), grouped by the sum of the other scores. This plot makes it clear that fitting a line for each question is appropriate even though the y=0.2x solution appears to work fine for one of the questions.

In this article I shared evidence that we should use different formulas for estimating missing responses in quality of life and care experience surveys. This is because the questions that can be marked “not applicable” have different average scores than other survey questions. Getting this right matters because these surveys are part of Australia’s mandatory quality indicator system. More accurate measurement means the government, aged care providers, and researchers have better data about residents’ actual experiences. The good news is that there’s a simple, data-driven solution. The formulas I’ve proposed here would provide significantly more accurate estimates for missing responses. When we’re measuring something as important as residents’ wellbeing and care experiences, accuracy matters. Small changes in how we handle missing responses can lead to meaningful improvements in how well we capture this feedback from aged care consumers in the future.

Footnotes

1. The data reported to the government is the number of aged care consumers in each category for QOL-ACC and QCE-ACC, stratified by completion mode (self-completion, interview, or proxy).

2. Since the categories were made for integers we have to assume how these ranges would map to a continuous scale. For example, a score of 21.6 would be “Good” if 19-21 is interpreted as [19,22) or “Excellent” if 19-21 is interpreted as [19,21.5). In my notation, “[” bracket means that the 19 is included in the range while the “)” means the second value is not. The Department of Health and Aged Care have clarified that for the purpose of categorisation the number should be rounded to the nearest integer first, which is equivalent to the latter interpretation.

3. For this to technically be correct we require that the data is missing at random (MAR). This means that the probability of a missing response is only related to the observed data.

4. Based on data from April 2024 to March 2025

5. Technically we minimise the squared residuals.

6. As measured by the ratio of RMSE to a saturated model.

7. Unless we change the predictors at least.