5. Study design and methodology

5. Study design and methodology 

5.1 Challenges in evaluating new income management

There are a number of conceptual and practical challenges to evaluating this measure.

5.1.1 Conceptual challenges

Attribution

Separating the impacts of NIM from those of other policies and programs is challenging. NIM is being implemented as part of a range of intersecting Commonwealth and Territory social policy initiatives which will have an impact on individuals and communities. Examples include policies related to alcohol restrictions, school nutrition and attendance programs and a range of other health initiatives. There may also be changes to other policies during the evaluation period.

From an evaluation perspective a central challenge will be to differentiate the impact of income management from these other programs and interventions, as well as identifying both positive and negative interdependencies between these.

Nature of expected changes

NIM has a number of short, medium and long-term objectives at the individual and population levels and it will be very difficult to disaggregate these different outcomes at different levels.

Service availability and quality

NIM is predicated on the assumption that participants will use their income to benefit their children and improve their lives.  In order to do so they must avail themselves of a range of services and opportunities including purchasing healthy food, sending their children to school and having their health checked, attending financial counselling, attending TAFE and seeking work.  However if there is limited or no availability of some of these services or opportunities then this will undermine the effectiveness of the model, and will greatly reduce the likelihood of positive outcomes. The quality of services, including the skills and qualifications of workers and level of adherence to policy guidelines, may also impact on service outcomes. The identification of service delivery gaps, however, may prove to be a useful finding within itself, in terms of informing future policy development. The evaluation will therefore need to separate the effects of income management itself from the effects of the services associated with the model, in particular financial counselling.

5.1.2 Practical challenges

A significant number of individuals being income managed will be vulnerable and it may be challenging to engage them in the evaluation.

Data will need to be collected from income management participants from diverse backgrounds and living in very different areas (e.g. cities, town and remote communities) and the data collection instruments will need to be able to cope with the diversity of those being income managed.

• Data will need to be collected from those living in remote communities and those delivering services in remote communities. This creates logistical challenges.
• A substantial proportion of those being income managed (particularly in the Northern Territory) will be Indigenous. There are particular issues and challenges in collecting information from Indigenous Australians. The research ethical issues involved are discussed in Section 7.

A number of further complexities should be noted. As described in Section 2, there are four major groups directly affected by the new income management:

• those who are compulsorily managed because of the type of benefit they are receiving and their duration of benefit receipt (participation/parenting stream)
• those who are managed because a Centrelink social worker believes them to be particularly vulnerable (vulnerable stream)
• those who are referred by child protection authorities (child protection stream), and
• those who volunteer for income management (voluntary stream).

There are a number of different pathways people can take between the different types of income management. For example, people could move from Child Protection of Income Management (CPIM) to Voluntary Income Management (VIM) and VIM to CPIM. See Figure 1 on page 7 for more detail.

In addition, those affected can be categorised in relation to their previous exposure to income management. There will be people who were income managed under the NTER, because of the location in which they live, and those who are being income managed for the first time. Additionally, in the non-income managed population there will be people who have never had their income managed, as well as those who were previously managed but are no longer subject to the measure, such as Age Pension and Disability Support Pension recipients living in locations where income management was previously applied.

It is also important to note that the schematic program logic shown above could be expected to differ between the three main groups identified earlier. That is, the program logic for people who are income managed because of concerns about child neglect will differ from the logic for people who are income managed because of the type of benefit that they receive and their duration on that benefit. Both of these program logics will differ from that for people who volunteer for the program. It seems plausible that people who volunteer for the program will be more motivated to engage with income management and, therefore, are likely to have better outcomes than people whose incomes are compulsorily managed. Similarly, it might be anticipated that people who are income-managed due to referrals from child protection authorities may well have less favourable outcomes.

In considering the program logic outlined above it is also essential to bear in mind that outcomes for those who are income managed will reflect much more than the effects of the program. The overall context in which income management occurs is of crucial importance. General economic circumstance such as changes in unemployment can have major effects on outcomes for income-managed individuals. Similarly, if fresh fruit and vegetables are simply too expensive to be met out of current benefit levels for NT residents living in remote areas, or if housing costs are too high in some locations to be adequately met with existing rent assistance, then outcomes could well appear negative even if the program itself actually had a positive impact. In addition, there are likely to be other changes in policy from the Commonwealth or the State or Territory government that could impact on outcomes for people who are income managed and the communities in which they live, for example, unrelated changes in benefit policy, health policy, education policy, housing policy or child protection policy.

A further complicating factor is that individuals who are income managed under the new model will have a range of differing family circumstances that will affect the outcomes of the model. In some cases families or wider kinship networks may pool resources to assist those who are income-managed, which could either reinforce or undercut the objectives of the model, while other individuals may not have this form of family support.

These considerations suggest that it will be necessary to take a broad approach to defining outcome variables – that is, the preferred evaluation approach should identify both the key outcomes that are intended by the new income management but place these in a broader context that can capture the impact of changes in the general environment and other policy changes.

The design will use both qualitative and quantitative methods to answer the research questions. It is not possible to use an experimental design, and therefore the outcomes will have to be determined by triangulating data from different sources.

5.2 Data sources

An important and necessary component will be the collection of data needed to support the evaluation. A thorough data audit, conducted by the consortium, indicates that significant data gaps exist.  Many of those datasets that do exist are either not very reliable, not easily available for small geographic areas or are difficult to access for various reasons. 

The evaluation should utilise a variety of data sources. This reflects the particular characteristics of the program and the outcomes that are being measured. The use of a diverse set of data sources also allows the evaluation to be conducted in a multi-layered way, taking account of the reported experience of individuals, administrative information on this, and the perspective of those involved in the implementation of the program. It also permits the ‘triangulation' of particular outcomes which may be difficult to measure. A mixed-methods evaluation is proposed that draws together information from multiple sources.

The evaluation of NIM will need to draw on data from a number of sources. This section provides an overview of these sources. The data that are needed fall into four types:

• administrative by-product data (also termed system data) from governments
• system data from private enterprises (eg. retail sales)
• purpose-designed data collected from people on NIM, those involved in implementing the model and the broader community,  comprising both:
  • individual surveys utilising a range of approaches appropriate to the circumstances of different groups, and
  • qualitative data collection through interviews, focus groups and other mechanisms.
• existing survey data.

5.3 Timing of different components of the evaluation

The evaluation should be undertaken in two stages. Stage one is the development of the evaluation framework including the scope and methodology of the evaluation, and establishment of the methods and data collection. Stage one includes an early implementation snapshot study of service providers in the NT to establish their readiness to implement NIM, as well as surveys of income managed clients to capture benchmark data that reflects circumstances of individuals, families and communities soon after the implementation of NIM.

Stage two will include two sub stages; the first stage will involve providing a process evaluation report to FaHCSIA by December 2011.  This report will focus on the implementation of NIM and the barriers and facilitating factors to implementation.  It will also include the views of a range of stakeholders and indications of short term outcomes including transitions to IM, exemptions, service availability and store data. 

The second sub phase will focus on the short, medium and, where possible, longer  term impacts of NIM on people, their families and communities. Intermediate evaluation reports that synthesise results to date and inform future analysis should be delivered to FaHCSIA by the end of 2012 and 2013. The final outcome evaluation report should be provided by end of 2014.

This graphic shows the deliverables due throughout the project, along with the activities that will occur at each stage.  The deliverables are:

• December 2011 = Intermediate Evaluation Report (process)
• December 2012 = Intermediate Evaluation Report (Process & short term outcomes)
• December 2013 = Intermediate Evaluation Report (Short & medium term outcomes)
• December 2014 = Intermediate Evaluation Report (Final Outcome Evaluation Report

Ongoing data collection/information gathering will occur throughou the project.

Implementation Snapshop study will occur between August 2010 and December 2011.

Other inputs as required will occur from December 2011 until Decenber 2014.

5.4 Geographic analysis

We recommend that a key component of the impact evaluation be an ecological analysis describing the association between the prevalence of income management and key outcome variables aggregated across small to medium geographic areas.

Whether this analysis will be suitable for the examination of long-term outcomes will depend upon the extent to which people move between different locations. An examination of Centrelink administrative (and possibly Census) data on mobility patterns will thus need to be undertaken as a complementary component to this analysis.

The main motivation for this is that information on many of the key outcome variables such as expenditure patterns are difficult to collect at the individual level for those participating in the program and even more difficult to collect for comparable people not participating (or for participants prior to their participation). Moreover, there is intrinsic interest in community level outcomes.

The methodology proposes that aggregate information be collected for regions across the NT (and possibly other States) around the implementation period, including on:

• the proportion of the population (or some relevant sub-population) participating in NIM (or some aspect of NIM)
• outcome variables (e.g. expenditures, crime rates, child wellbeing outcomes) and
• confounding variables (such as the operation of other programs).

The correlation between the NIM participation rate and the outcome variables is examined while controlling for confounding variables. If data over time are available, fixed-effect models can be used which examine the changes in NIM and outcome measures in each region.

The main threat to the validity for this analysis, as with all non-experimental analyses, is that there may be unobserved differences across regions which are correlated with both the outcome variables and NIM participation. For example, areas which have a high proportion of the population moving onto NIM might also experience a large increase in other interventions at the same time. If this is not also measured and controlled for the observed association will be a biased estimate of the impact of NIM on outcomes.

Similarly, the cessation of the old model of income management will need to be controlled for (or might form an intervention variable to be analysed in its own right). The associated threat to the reliability (or precision) of such an analysis is that, once all these potential confounders are controlled for, there may be insufficient independent variation in the NIM participation variables to enable comparison of different levels of NIM. Whether this will be the case is difficult to ascertain prior to data collection. A necessary requirement for the geographic analysis described here to be informative is that there be sufficient geographic variation in the changes in NIM participation over time. The first exploratory stage of the geographic analysis would thus be to analyse the geographic spread of income management participation patterns using the Centrelink administrative data.

For example, even if the NIM is rolled out at the same time to all regions of the NT, there will be some regions where a large proportion of the population is subject to this program, and other areas where the proportion subject in the population is small. If favourable changes in the outcome variables are observed in the former areas but not the latter, then this will provide strong evidence on the efficacy of the model.

Because most components of the NIM will vary together at the regional level, this impact analysis will be most suited to measurement of the overall impact of the NIM program, rather than particular components.

Note that it is intrinsically impossible in this analysis to separate the impact of the NIM model from other variables which vary closely along with it. For example, NIM is targeted at particular disadvantaged groups. In the case described in the previous paragraph, one cannot rule out that the observed association will be due to these groups doing better for some unexplained reason or because of another intervention. The research can only be made more robust by seeking to understand the impact of all the potential confounding factors.

The following considerations should guide the collection of data for this exercise.

5.5 Geographic scope

Ideally, the scope for this exercise should include comparable areas outside of the NT which have not been subject to IM. Data extraction from Commonwealth data collections should be designed with this intention in mind. However, many of the key outcome variables are only available via State government departments. These variables may be both defined differently and available for analysis in different forms in different States – which might thus require a restriction to the NT. Nonetheless, if it is envisaged that NIM will be generalised to other states and territories, collecting data from these jurisdictions now may form the grounding for future evaluations of those programs.

5.6 Time scope

The data should preferably cover the time period starting several years before the NIM implementation to a period after NIM is well-established and bedded down.

5.7 Regional aggregation

The data for the outcome variables should be collected at as small a regional level as possible, consistent with the relevant catchment areas for the different outcomes. For example, for alcohol expenditure, a suitable unit might be a township or a suburb. In both cases there might be spill-over effects into adjoining regions, but this can be modelled in the data analysis if the initial data collection is at a suitably low level of aggregation.

Similarly, data on migration between the regions can be incorporated into the modelling exercise. The most useful data for this will be Centrelink data on location patterns of the whole client base – not just those involved in NIM.

The different outcome variables will generally be available at different levels of aggregation. It is therefore important that the geocoding in the NIM data be as detailed as possible so as to permit the creation of NIM participation estimates at levels of aggregation that match the different outcome variables.

5.8 Take-up and participation in new income management

If geo-coded data on NIM participation is available, this can then be compared with Census and other estimates of small area populations to estimate the NIM participation rate in each area over time.

5.9 Confounding factors

The key confounding factors to be considered will be the presence of other policy interventions in the different areas. Detailed information on these will need to be included in the modelling. See Appendix G: List of funded initiatives in the NT that support vulnerable children and families.

5.10 Sub components

As NIM contains four types of participants (mainstream, vulnerable, voluntary and child protection), the framework will seek to address each group separately.  This is particularly true for the child protection component which not only potentially serves a different group of people, but also has very different entry and exit processes.  Although the data for the sub-components will overlap, it is important to disaggregate these components in the analysis to ensure that the appropriate processes and outcomes for each group are treated separately.

5.11 Evaluation of the child protection measure in NT

Evaluation of CPIM will require some specific data collection. Data will need to be collected from child protection workers in the form of either in-depth interviews or focus groups.

It is also proposed that a case file review be undertaken and coded according to a pro forma. This methodology is particularly effective for creating de-identified data from confidential client files.4 It is proposed that a case file review be conducted to evaluate the impact of CPIM on child protection outcomes (e.g. re-notification, re-substantiation, substantiated type abuse and primary presenting problems).

The evaluation will examine data provided by the NT Government, which will track those families who have been referred to CPIM.  The NT Department of Health and Families will provide information on:

• incidents of child protection notifications
• category of child protection notification (e.g.  neglect, physical abuse, sexual abuse or emotional abuse)
• whether or not the notification resulted in investigation by a caseworker
• whether or not reports were substantiated
• broad identifier of the reporter type (e.g.  hospital, family, policy, school) and
• number of the child's encounters with youth justice system.

The Child Protection records of each child whose family is referred will also be examined for up to a 5-year period preceding the referral, in order to establish whether income management results in changes in re-notification rates for families. Information will also be collected about the service use of these families including:

• What other support services has the family been referred to?
• What other services has the family accessed or failed to access?

The NT Department of Health and Families (DHF) caseworkers will also be surveyed via online electronic surveys which can be completed in various sessions over a period of time (e.g. several weeks) to allow for workload management. These surveys will be similar to those of Centrelink workers and other stakeholders, and will cover their attitudes to IM, training, relationships with other agencies and barriers and facilitating factors to helping families who neglect their children.

In addition to these data about the families, more qualitative information will also be sought via the child protection caseworkers, including:

• children's access to adequate food, clothing, education, health services, notifications and stability of living arrangements and
• parents' attitudes, financial management skills and confidence, levels of stress, knowledge of IM, exposure to harassment.

4. AIFS has successfully used this approach in evaluating the Magellan program and the 2006 changes to the family law system.