Salvation Army questions reliability of government data and targets
Tue 23 Feb 2016
The Salvation Army released its 9th annual State of the Nation report on 17 February 2016. The reports examine statistics on social and economic ...
The reports examine statistics on social and economic conditions, including child abuse and neglect and other forms of family violence.
The report is critical about the Government's Better Public Services targets. These include targets to reduce the number of physical assaults of children by 5% and reduce the number of violent crimes committed by 20%.
The 2016 State of the Nation report is entitled Moving Targets. The author, Salvation Army social policy analyst Alan Johnson, states,
"We see this as a dual metaphor. On the one hand, it is an allusion to the way in which some government agencies appear to be using targets and the numbers behind them in a less than straightforward and reliable manner. This practice has many subtle and ingenious approaches—ranging from changing the definitions behind indicators so results appear better, to inventing new numbers like pseudo-service level indicators that are difficult to verify and have no point of reference, to simply changing reporting formats in the name of improving accountability without any improvement in the reliability of the information provided. All of these approaches have recently slipped into the reporting practices of some government agencies.1 While the targets may not change with such practices, the meaning behind them can. This can cause us to slip into a 'movable feast' mentality, where we find any reason to celebrate success or progress even though we have lost our sense of the purpose behind it all."
The author also notes:
"Given the political capital the Government has invested into these result areas and targets, there is huge pressure on public sector managers to come up with favourable results. And come up with favourable results they do, as indicated in the latest results reported by the State Services Commission.2 The problem is the people and agencies responsible for the results are also largely responsible for the analysis and reporting of them."
The author notes that the Ministry of Social Development's 2015 Annual Report intimated that substantial progress was being made in reducing the number of child abuse and neglect cases. However, the State of the Nation report asserts:
"Given that the number of child abuse/neglect cases investigated or substantiated are administrative figures determined by practice, there is no way of knowing if the background level of abuse or neglect is falling, rising or staying the same. ... It is certainly the case that the number of children in CYF care has remained almost constant since 2011 (at around 5,000 children), so it does not appear the numbers of very serious cases of neglect or abuse have subsided at all."
In relation to violent crime, the report makes comment on the new Police dataset on victimisation, introduced in 2014. This dataset includes information on the relationship between the victim and offender in criminal offending, theoretically making family violence cases easier to identify. However the report highlights that in a very high percentage of cases, the relationship between the victim and offender is not being recorded:
"For assaults, for example, the relationship between the offender and victim is not recorded in 53% of cases. It seems remarkable that over half the victims of violence who come to Police attention are unsure of whether they know their assailant or not."
On the new Police dataset, the report concludes:
"While the emphasis placed on victims of violence in the new reporting approach should be applauded, the obvious and major flaws in the information offered by this approach give it little policy or operational value. The need for a more comprehensive approach to gathering and reporting data on domestic violence remains and it seems unlikely that current efforts will produce reliable information."
The New Zealand Family Violence Clearinghouse has also highlighted concerns with using Police or other administrative data to monitor trends, and recommended ways in which data collection needs to be improved. Administrative data is affected by changes in organisations' policies and procedures and accordingly, cannot be considered a reliable source of data for monitoring trends in family violence in the community over time. For further information, see:
Gulliver, P., Fanslow, J. (2012). Measurement of family violence at a population level: What might be needed to develop reliable and valid family violence indicators? Auckland: New Zealand Family Violence Clearinghouse, University of Auckland
Further commentary and discussion on the State of the Nation report is available in the media stories below.
The Salvation Army's 2015 State of the Nation report included critical comment on falling crime rates.
Image: 3D Bar Graph Meeting by Scott Maxwell Licence: Creative Commons Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)