Are We Measuring What We Think We Are Measuring?

What a privilege to attend the Extreme Teamwork & Decision-Making Mini Conference at the University of Liverpool.

The conference brought together people from across the blue light services to explore decision making in high-pressure and extreme situations. Coming from a very different professional world, I was struck by how many of the themes resonated with children’s social care.

Decision making under pressure. Uncertainty. Conflicting information. Cognitive load. Trust. Relationships. Expertise. The consequences of getting decisions wrong. And, running through it all, the importance of keeping the person at the centre.

In children’s social care, that person is the child.

Professor Paul Christiansen’s session on statistics and accurate measurement in research resonated with me.

Complicated things are hard to measure. You can end up measuring the wrong things.

Children’s social care policy making relies heavily on data. Numbers are used to describe demand, placements, outcomes, costs, performance and the effectiveness of services. They influence policy, help shape decisions about what should change and what the public purse funds.

But data does not speak for itself.

The danger is not just measuring the wrong thing. It is asking the data to answer a question it cannot answer.

The same caution applies to the tools used to measure things in the first place. A questionnaire or scoring system can produce a neat number, but that does not make it evidence. If a measure has not been properly tested, we do not know whether it is reliable or whether it measures what it claims to measure.

For decades, poorer outcomes for children who have lived in residential care have been used to support a policy preference for fostering. The data can show a difference between groups. It cannot, on its own, tell us that residential care caused that difference.

Nor does it tell us that fostering would have been the right alternative for those children.

Fostering is not the problem, and residential care is not the problem. The problem is treating one form of care as inherently preferable to another, rather than asking what this child needs.

Every child’s history is different, and every journey through care is different. If those differences are overlooked, poorer outcomes can too easily be blamed on placement type rather than understood in the context of the child’s life. Policy then risks being built on the wrong conclusion, and children do not get the care they need.

Knowing how many placements a child has had tells us about moves. We know moves can matter, but the number alone does not tell us what those moves meant for that child.

Knowing that a child entered residential care after foster care does not tell us what happened in the placements, whether foster care was right for that child, or whether their needs were met.

Outcome data tells us what happened. It does not, on its own, explain why it happened. If a cause or contributing factor is claimed, there needs to be evidence to support that conclusion.

Research can too easily be used to support a belief, rather than to test what the evidence actually shows.

That matters when research is being used to influence policy that affects children’s lives.

Professor Catherine Durose’s session on co-production added another dimension.

It is not only about whether something has been measured well, but who decided what mattered enough to measure in the first place.

Whose knowledge shaped the question? Who was involved in interpreting what was found? And who was missing?

Co-production is not simply asking people for their views once the important questions have already been decided. At its best, it allows people with different forms of knowledge and experience to help shape the question itself.

That feels particularly important in children’s social care.

Children, families, caregivers and practitioners know things that are not visible in an administrative dataset. Their knowledge does not replace research. It makes it more meaningful.

Throughout the conference, the similarities with children’s social care kept appearing. There were discussions about risk, uncertainty, trust, communication, shared information, expertise, relationships and decision making when the stakes are high.

There was also an acknowledgement that the conditions in which people make decisions matter.

A good assessment does not create the service a child needs. Identifying the right placement does not mean that placement exists. Knowing what a child needs does not mean the system can provide it.

Decision making cannot be separated from the choices that are actually available.

In extreme settings the language is about saving lives.

In children’s social care, it is about protecting the child’s right to a good life.

That makes the quality and use of the evidence behind policy decisions important.

Perhaps some of the most useful questions are also the simplest:

Are we measuring what we think we are measuring?

Are we asking the data a question it can actually answer?

Does the evidence really support the conclusion being drawn from it?

And are the right people involved in deciding what should be measured in the first place?

Children’s social care is complex because children’s lives are complex. That is not a reason to give up on evidence or measurement. It is a reason to ask better questions, use evidence more carefully and be honest about what the data can and cannot tell us.