Salvation Army questions reliability of government data and targets

Tue 23 Feb 2016

The Salvation Army released its 9th annual State of the Nation report on 17 February 2016. The reports examine statistics on social and economic ...

The Salvation Army released its 9th annual State of the Nation report on 17 February 2016.

The reports examine statistics on social and economic conditions, including child abuse and neglect and other forms of family violence.

The report is critical about the Government's Better Public Services targets. These include targets to reduce the number of physical assaults of children by 5% and reduce the number of violent crimes committed by 20%.

The 2016 State of the Nation report is entitled Moving Targets. The author, Salvation Army social policy analyst Alan Johnson, states,

"We see this as a dual metaphor. On the one hand, it is an allusion to the way in which some government agencies appear to be using targets and the numbers behind them in a less than straightforward and reliable manner. This practice has many subtle and ingenious approaches—ranging from changing the definitions behind indicators so results appear better, to inventing new numbers like pseudo-service level indicators that are difficult to verify and have no point of reference, to simply changing reporting formats in the name of improving accountability without any improvement in the reliability of the information provided. All of these approaches have recently slipped into the reporting practices of some government agencies.1 While the targets may not change with such practices, the meaning behind them can. This can cause us to slip into a 'movable feast' mentality, where we find any reason to celebrate success or progress even though we have lost our sense of the purpose behind it all."

The author also notes:

"Given the political capital the Government has invested into these result areas and targets, there is huge pressure on public sector managers to come up with favourable results. And come up with favourable results they do, as indicated in the latest results reported by the State Services Commission.2 The problem is the people and agencies responsible for the results are also largely responsible for the analysis and reporting of them."

Child abuse

The author notes that the Ministry of Social Development's 2015 Annual Report intimated that substantial progress was being made in reducing the number of child abuse and neglect cases. However, the State of the Nation report asserts:

"Given that the number of child abuse/neglect cases investigated or substantiated are administrative figures determined by practice, there is no way of knowing if the background level of abuse or neglect is falling, rising or staying the same. ... It is certainly the case that the number of children in CYF care has remained almost constant since 2011 (at around 5,000 children), so it does not appear the numbers of very serious cases of neglect or abuse have subsided at all."

Violent crime

In relation to violent crime, the report makes comment on the new Police dataset on victimisation, introduced in 2014. This dataset includes information on the relationship between the victim and offender in criminal offending, theoretically making family violence cases easier to identify. However the report highlights that in a very high percentage of cases, the relationship between the victim and offender is not being recorded:

"For assaults, for example, the relationship between the offender and victim is not recorded in 53% of cases. It seems remarkable that over half the victims of violence who come to Police attention are unsure of whether they know their assailant or not."

On the new Police dataset, the report concludes:

"While the emphasis placed on victims of violence in the new reporting approach should be applauded, the obvious and major flaws in the information offered by this approach give it little policy or operational value. The need for a more comprehensive approach to gathering and reporting data on domestic violence remains and it seems unlikely that current efforts will produce reliable information."

The New Zealand Family Violence Clearinghouse has also highlighted concerns with using Police or other administrative data to monitor trends, and recommended ways in which data collection needs to be improved. Administrative data is affected by changes in organisations' policies and procedures and accordingly, cannot be considered a reliable source of data for monitoring trends in family violence in the community over time. For further information, see:

Gulliver, P., & Fanslow, J. (2013). Family violence indicators: Can administrative data sets be used to measure trends in family violence in New Zealand? Wellington: Superu

and

Gulliver, P., Fanslow, J. (2012). Measurement of family violence at a population level: What might be needed to develop reliable and valid family violence indicators? Auckland: New Zealand Family Violence Clearinghouse, University of Auckland

Further commentary and discussion on the State of the Nation report is available in the media stories below.

Background information

The Salvation Army's 2015 State of the Nation report included critical comment on falling crime rates.

Media

Presenting the rosiest view a simple survival mechanism in politics, Editorial: Stuff, 18.2.2016

Child abuse data doesn't tell full story – Sallies, Radio NZ, 17.2.2016

Is CYF manipulating child abuse statistics? (interview with Nicola Atwool), Radio NZ, 17.2.2016

Is CYF manipulating child abuse statistics? (interview with Paul Nixon, CYF Shief Social Worker), Radio NZ, 17.2.2016

CYF denies massaging abuse statistics, Radio NZ, 17.2.2016