Using data to improve

There are four types of measures necessary for monitoring and improving systems such as schools and classrooms. These are:

Performance measures
Perception measures
Process measures
Input measures.

Performance and perception measures are generally well understood within schools; not so process and input measures. This post seeks to illuminate the use and significance of each type of measure. It is an edited extract from our forthcoming book IMPROVING LEARNING: A how-to guide for school improvement.

Performance Measures

Performance measures are defined to be:

measures of the outcomes of a system that indicate how well the system has performed.

Performance measures relate to the aims of the system. They are used to quantify the outputs and outcomes of the system. In this way, performance measures relate to the key requirements of stakeholders as reflected in the aims of the system.

Examples of performance measures in a school include:

student graduation and completion rates
student results in national, state and school-based testing
expenditure to budget
student and staff attendance.

Performance measures answer the question:
how did we go?

They are collected and reported for three reasons:

To understand the degree to which the aims of the system are being met
To compare performance of one system with that of another similar system
To monitor changes in performance over time.

Performance measures have two major deficiencies: they report what has happened in the past; and they generally provide no insight into how to improve performance. For these reasons, we need other types of measures as well.

Perception Measures

Perception measures are defined to be:

measures collected from the stakeholders in the system in order to monitor their thoughts and opinions of the system.

Perception measures provide insights into peoples’ experience of the system. Given that people make choices based on their perceptions, whether these are accurate or not, perception measures provide valuable insights that can be useful in explaining and predicting behaviour.

Perception measures are collected and reported for three reasons:

To understand the collective perceptions of key stakeholder groups
To identify perceived strengths and opportunities for improvement
To monitor changes in perceptions over time.

For schools, it is important that data are collected and reported regularly regarding the perceptions of staff, students and families. This data can include:

opinions regarding the school’s services
satisfaction with the school and its operation
thoughts and opinions about specific aspects of the school.

Perception measures answer the question:
what do people think of the system?

Note that care is required to ensure adequate random samples are collected for perception data to be reliable.

Process Measures

Process measures are defined to be:

measures collected within the system that are predictive of system performance and which can be used to initiate adjustments to processes.

Process measures are used to monitor progress and predict final outcomes. Most importantly, process measures are used to identify when changes are required in order to bring about improved performance.

Examples of process measures in schools include:

progressive student self-assessment of knowledge and skills, such as ongoing self-assessment using a capacity matrix
practice tests
teacher assessment of student progress
home learning and assessment task completion rates
monthly financial reports.

Notice that at the classroom level, process measures relating to learning are also known as ‘formative assessments’ – assessment used to inform the learning and teaching processes. These short, sharp and regular assessments are used to identify what to do next to improve the students’ learning progress.

Process measures are collected and reported regularly. This enables intervention when a process appears to be at risk of delivering unsatisfactory outcomes — before it is too late. With student learning this is critical so that mediation — improvements to learning — can be made, ensuring that high levels of learning are maintained.

Process measures answer the question:
how are we going?

Input Measures

Input measures are defined to be:

measures collected at the boundary of the system to quantify key characteristics of the inputs that affect system operation.

Inputs to a system affect system performance. In the manufacturing sector, the key characteristics of important system inputs – such as material thickness or lubricant viscosity – can be controlled. Other key process input variables cannot be controlled and process adjustments are necessary based on measurements of these inputs. For example, farmers cannot control rainfall, but they can adjust their rates of irrigation based on rainfall measurements.

Teachers cannot control the prior learning of their students, but they do adjust their classroom processes based on this input characteristic. Similarly, very few schools can control the value that their students’ families place on education, but they can adjust school processes based on data about this.

Input measures — data relating to critical characteristics of key inputs to a system — are required to ensure that appropriate actions are taken within the system to accommodate changes and variation in inputs. They are also required if systems are to become robust to input variation.

The core process in a school is learning (not teaching). The learning process is subject to enormous variation in inputs. Key input variables include students’ prior learning, motivation to learn, family background, home support and peer pressure, to name a few.

Any classroom of students will display enormous variation in these inputs. It is not uncommon, for example, to have a class of 13-year-olds with chronological reading ages varying from seven to 18 years. Understanding input variation is crucial to designing learning processes that cater to the many and varied needs of all students.

At a whole school level, there is variation among teachers. Teachers’ knowledge of the content they are required to teach is not uniform, nor is their knowledge of students’ learning processes or the programs in use at the school. Experience with school and education system compliance requirements can also be highly variable. School processes need to take account of this input variation.

Like process measures, input measures are used to make adjustments to system and process design and operation in order to ensure consistently high performance.

Input measures answer the question:
what are the key characteristics of the inputs to the system?

To recap, there are four sets of measures that can be used for monitoring and improving systems:

Performance measures
Process measures
Perception measures
Input measures.

Taken together, these measures provide deep insight into the performance, operation, and behaviour of a system. These measures provide a voice with which the system can speak about its behaviour, operation and performance. Importantly, these measures support analysis and prediction of future system behaviour and performance.

The four sets of system measures can be collected for a whole system as well as for subsystems within it. In school systems, this means that these data can be collected at the state, region/district, school, sub-school, classroom, and student levels.

Watch a video that describes these four types of measures.

Purchase our Learning and Improvement Guide: Using Data to Improve.

Learn more about our forthcoming book, IMPROVING LEARNING: A how-to guide for school improvement.