4. New income management evaluation framework

This section outlines the consortium's proposed evaluation framework. It is expected that some elements of this framework will be further developed by the independent evaluator as the project progresses.

This framework addresses scope of assessment, high-level research questions, study design, methodologies, proposed data (and sources) and an indicative timetable. Research Instruments and more detailed evaluation questions will be developed in Phase 2 of the evaluation project.

The main purposes of the evaluation activities to be conducted over the 2010-2014 financial years are to:

• provide evidence on the impact on NIM on those who are affected
• assess whether the reforms were implemented effectively, and
• understand whether NIM is a cost-effective model so as to inform future government decision making and social policy formulation for both the wider and the Indigenous communities.

4.1 Scope of assessment

The evaluation should contain a process evaluation, an outcome evaluation and a value for money analysis. This evaluation strategy has been informed by a comprehensive literature review of evaluations of similar conditional welfare programs internationally (see Appendix E).

4.2 Process evaluation

The process evaluation should examine how NIM was implemented and should report on the barriers and enablers which affected its implementation. (See Section 5 for more detail).

4.3 Outcome evaluation

The outcome evaluation should assess the impact of the model at the individual, family/household and community level in the Northern Territory. The evaluation should draw on administrative data, survey data, longitudinal interviews and case studies to determine the extent of individual, family/household and community level changes over 2010-14 financial years.

Regular reports should be delivered to FaHCSIA by end of each calendar year from 2011-2014, including:

• a substantial progress report addressing implementation issues and early progress in achieving short term outcomes, by the end of 2011
• annual intermediate evaluation reports that synthesise results to date and inform future analysis, by the end of 2012 and 2013, and
• a final outcome evaluation report  by the end of 2014 (See Section 5).

Interim evaluation reports (short-medium term outcome evaluations) should be conducted on an annual basis and be delivered by end December 2011-2013 – see Sections 5 and 6 for more detail. The final evaluation report will be delivered by the end of December 2014 and contain information on the short-medium, and wherever possible longer-term, outcomes along with value for money analysis3. See Section 4.4.2 for more detail.

With regard to the evaluation questions and indicators, the evaluation will seek to provide answers to the following high level questions. At this stage, the questions are broad and there is a need to further develop specific research questions in Phase 2 of this project.

[ top ]

4.4 Broad overarching questions across all four streams of new income management (ie participation/parenting, child protection, voluntary and vulnerable streams)

4.4.1 Process evaluation

1. How effectively has NIM been administered and implemented?

• What have been the resource implications of implementing the program?
• Have suitable individuals and groups been targeted by NIM?
• Have people been able to transfer into and out of NIM appropriately (e.g. choosing to transfer from income management under NTER to VIM)?
• What has been the effect of the introduction of NIM on service providers?

2. What is the profile of people on the different income management streams?
3. Have there been any initial process 'teething issues' that need to be addressed?
4. What are the views of participants in the NIM model and their families on the implementation of the program? 
5. Has the measure been implemented in a non-discriminatory manner?

4.4.2 Outcome evaluation

1. What are the short, medium and longer-term impacts of income management on individuals, their families and communities?

• How do these effects differ for the various streams of the program (mainstream, voluntary, child protection, vulnerable)?
• Have there been changes in spending patterns, food and alcohol consumption, school attendance and harassment?
• What impact does NIM have on movement in and out of NT among people on the measure?
• Has NIM contributed to changes in financial management, child health, alcohol abuse, violence and parenting (ie reduced neglect)?
• Do the four streams achieve appropriate outcomes for their participants?
• Has NIM had any unintended consequences (positive or negative)?
• Are there differential effects for different groups? (including—if sufficient data is available—by Indigeneity, gender, location, age, educational status, work status, income, length of time on income support, marital status, family composition and diverse cultural and linguistic background)
• Does IM provide value for money by comparison with other interventions?
• Does NIM provide any benefits over and above targeted service provision? 

[ top ]

4.5 Research questions for specific streams of the NIM model

4.5.1 Questions specific to the participation/parenting stream

1. Has NIM helped to facilitate better management of finances in the short, medium and long term for people on income management and their families?
2. Has access to services or interventions improved for those families?
3. Have other changes in the wellbeing and capabilities of the individuals and families occurred?

4.5.2 Questions specific to the child protection stream

1. What has been the impact of income management on child neglect?
2. What has been the impact on child wellbeing in those families referred to the child protection measure (CPIM)?
3. What are the barriers and facilitating factors for child protection workers to use income management as a casework tool?
4. What (if any) service delivery gaps have impacted on the usefulness of the CPIM?

4.5.3 Questions specific to the vulnerable stream

1. Are vulnerable people appropriately targeted by this measure?
2. How does income management impact on the vulnerability of individuals?
3. Have people on this stream experienced changes in the level of harassment (e.g. humbugging)?

4.5.4 Questions specific to the voluntary stream

1. Have people who volunteered for income management been able to make an informed choice?
2. How long do voluntary income management recipients stay on the measure?
3. What are the key motivations for people who voluntarily access income management, and why do they stop?
4. An analysis of value for money should assess the cost effectiveness of new income management. The cost effectiveness analysis will use the quantitative data from the outcome evaluation and financial data from programs to assess the extent to which the costs produced tangible benefits.