Evaluation Framework for New Income Management (NIM)

December 2010 

This outlines a framework for the evaluation of the NIM model in the NT. The framework is intended to provide a broad structure for the evaluation. It addresses the scope of assessment, high-level research questions, study design, methodologies and proposed data and sources.

Executive summary 

In May 2010, a consortium of experts from the Social Policy Research Centre (SPRC), the Australian Institute of Family Studies (AIFS) and the Australian National University (ANU) was engaged by the Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA) to develop an overarching framework for the evaluation of new income management (NIM) to guide evaluation activities over the period 2010-14.

The terms of reference for developing the evaluation framework are that the evaluation activities:

• be completed and reported by December 2014
• provide information on the implementation of the NIM in the Northern Territory by the end of 2011 in order to inform decisions about an expansion of the model beyond the Northern Territory
• result in data being collected that can be used to evaluate short, medium and, where possible, longer-term impacts/outcomes of new income management
• include a set of ethics guidelines and an ethical clearance strategy relevant to this evaluation project.

This document outlines a framework for the evaluation of the NIM model in the NT. The framework is intended to provide a broad structure for the evaluation. It addresses the scope of assessment, high-level research questions, study design, methodologies and proposed data and sources. (See Section 4 for more detail.)

Undertaking the evaluation will require an iterative approach and, as such, it is expected that some elements of this framework (including the program logic, the design of research instruments and detailed questions) will need further development as the project progresses.

The development of the evaluation framework involved extensive consultations with both government and non government sectors. The evaluation framework takes account of:

• the way in which NIM is being implemented, especially that it has been rolled out to the whole of the Northern Territory in a short period of time
• the previous income management policy which has been operating in the Northern Territory since 2007
• the fact that other policies may change at the same time as NIM is being implemented
• practical challenges involved in collecting data in remote areas of the Northern Territory
• particular ethical issues involved in collecting data from vulnerable groups including: children, women, the elderly, Indigenous Australians, and people from culturally and linguistically diverse backgrounds.

It takes a multi-method approach (also called triangulation) as it is not appropriate to use an experimental design on a complex social policy such as this. This means collecting different types of primary data (qualitative and quantitative), using secondary administrative and survey data, and collecting information on the same questions from multiple informants (e.g. those being income managed, Centrelink staff, financial counsellors, child protection workers).

As a first step, the framework involves collecting data for an early implementation snapshot, which will also provide benchmarking data for the evaluation. The consortium of researchers who developed this evaluation framework has been commissioned to collect the majority of the early implementation data.

The development of the evaluation framework and collection of the early implementation snapshot data constitutes Phase 1 of the evaluation. The conduct of a comprehensive and independent evaluation of the NIM constitutes Phase 2.

Phase 2 involves evaluating:

• the effectiveness of the program's implementation
• whether the program was delivered as intended to the target population in a fair and equitable manner (including access to necessary services)
• an assessment of the impacts of NIM on individuals, families and communities in the Northern Territory and
• an analysis of value for money (to the extent that this is achievable within the timeframe of this evaluation).

The evaluation is designed to produce the data necessary to evaluate short, medium and, where possible, the longer-term outcomes of NIM. The short term outcomes are foundations and enablers for the measure, and should be evident within the early years of the evaluation project. Behavioural changes/outcomes could occur in the medium term and be evident within the life of this evaluation project.  However, some of the longer-term and sustained changes in behaviour are likely to take a number of years to become evident.

All evaluation activities will need to continue to be undertaken in close consultation with both government and non-government stakeholders.

A requirement of the evaluation framework was that it allow the data to be produced to enable reports to be delivered to FaHCSIA by end of each calendar year from 2011-2014, including:

• a substantial progress report addressing implementation issues and early progress in achieving short term outcomes, by the end of 2011
• annual intermediate evaluation reports that synthesise results to date and inform future analysis, by the end of 2012 and 2013 and
• a final outcome evaluation report  by the end of 2014.