Appendix F: Data sources to support the evaluation

F1 Administrative data sources
 

F1.1  Centrelink administrative data

The main source of administrative data which will be used in evaluating NIM is Centrelink Management Information System (MIS) data. This will enable a comparison of individuals who are income managed with individuals who are not income managed, including in areas of Australia not subject to income management.

This data will provide information on:

• the implementation of NIM
• reasons for being subject to income management (disengaged youth; long-term welfare recipients; referred by child protection authorities; vulnerable welfare payment recipients)
• reasons for exemptions from income management
• the allocation of income managed income by areas of spending
• attendance at a money management and financial counselling service
• receipt of matched savings program
• movements into and out of income management by reason
• whether those being income managed continue to receive income support payments 
• for those who continue to receive some Centrelink payments, whether they are have income from paid employment and
• geographic mobility (for example if the introduction of NIM results in persons potentially impacted moving from the NT).

The Centrelink administrative data can provide longitudinal data on the experience of those being income managed and how they compare to those of people not being income managed. It also allows the experience and outcomes of different groups to be compared, including:

• Indigenous status
• geographic location and
• whether income management is voluntary or compulsory.

F2  Northern Territory administrative data

Consultations with NT government departments indicates that generally administrative data produced by the NT government does not provide information on whether people are receiving particular types of Centrelink payments or whether they are being income managed.

While this data cannot be linked to the specific individuals affected by NIM it can be used to evaluate the impact of NIM collectively.11 The key areas for which it appears that NT administrative data can contribute to the evaluation of NIM are:

• child health and development (low birth weight; stunting, wasting, underweight, anaemia in children under 5 12, child protection substantiations)
• education (school enrolment and attendance data13 ; National Assessment Program – Literacy and Numeracy (NAPLAN) data14 )
• sales of alcohol15 and tobacco16
• crime data17
• housing data and
• incidence of late payment/non-payment of utilities bills.

To the extent possible data is sought on a detailed geographic basis, and where possible as a time series. This reflects the analytical needs of the evaluation. Utilising the ecological approach, and comparing the outcomes for regions with different proportions of the population subject to income management, permits the isolation of the impact of the program, relative to general trends and other universal changes. Similarly the availability of time series data permits trends to be observed, as well a 'before' and 'after' data.  It is important to note that the evaluators will require permission to use all these datasets, and that some data may not be available (or the evaluation may not be given permission to use the data) for small communities.

F3 Store based data

There are sources of administratively based store data that will be of value to the evaluation. These are:

• information derived from the licensing arrangements for stores in remote communities and
• point of sale/scanner data on the composition of spending.

In addition, as detailed below it is proposed to utilise surveys to obtain more specific information from retail and other outlets.

F4 Evaluation specific data collections

Ideally, in evaluating social policies data would be collected prior to the implementation of the policy (baseline data). Data collected after the implementation of the policy would then be compared to the baseline data to track changes over time.

However, in the case of NIM, the fact that income management has already been happening in 73 discrete communities since 2008 makes collecting baseline data that that have not been tainted by the earlier, more localised, income management scheme difficult, if not impossible.

In order to evaluate NIM, primary data will need to be collected from:

• people subject to income management
• Community Elders
• Centrelink staff
• merchants
• money management and financial counselling service providers and
• child protection case workers.

A combination of quantitative and qualitative data will be needed. While NIM is being implemented across the NT it will be necessary to understand the extent to which the implementation and impacts of income management vary by gender, location, age, educational status, work status, income, length of time on income support, marital status, family composition, and diverse cultural and linguistic background.

F5 Survey of clients subject to income management

This component involves conducting randomly-selected representative quantitative surveys of people being income managed and those who had been income managed in the past (under the NTER). These surveys would be supplemented by a qualitative study which would provide a rich description of their attitudes towards and experiences of income management and the impact it had on them and their families.

The first survey would be of people being income managed at the time of the interview or likely to commence on income management in the near future. The second would be of those who had been income managed in the past but were no longer being income managed at the time of the survey.18

This would allow data to be collected from people who are subject to income management for an extended period (or who have returned to income management after a period of not being subject to income management) and those who have moved off income management.

While ultimately the sample size that can be achieved will depend upon cost, the sample size for each survey should ideally include at least 500 people. The sample needs to be randomly selected, and it should be stratified in order to ensure that data are collected from individuals of various ages, from men and women, and from a range of different geographical locations that represent the diversity of the NT. The areas selected should include major town/cities, regional centres and remote communities.

The survey will need to collect information from a representative sample of people subject to the measure in order to assess whether experiences and outcomes differ. The number of geographic areas from which data is collected will depend upon cost. The greater the number of areas from which data is collected the greater the cost. The sample should be clustered both to contain cost and to allow community level effects to be estimated. The sample should include those who are being compulsorily income managed and those who are being voluntarily income managed. The sampling frame will be from Centrelink administrative data on people on income management.

This kind of sample is very challenging to administer, particularly in remote areas, and the evaluators will need to test out appropriate methodologies for engaging evaluation participants and facilitating accurate data collection.

Attempts should be made to link the survey data of participants to other data sources including administrative data. This data linkage can only happen with consent and approval would be required from data custodians. Possible sources of data are enrolment and attendance at school for children, NAPLAN and data on child development and health.

The in-depth interviews will ask participants to consider the impact of income management on their families. Family impacts will be important from a number of perspectives; for example, it is possible that some individuals will be income-managed while others in their family will not, and alternatively, some income-managed individuals will be part of families where most family members are also covered by the scheme, with potential implications for outcomes. Family behaviours will also be of interest, since outcomes could be affected by whether people tend to make “whole of family” shopping trips or purchase items together or in bulk.

This component would collect information on a range of issues including:

• expenditure on and consumption of alcohol, tobacco and gambling
• parenting skills
• understanding of income management
• experience of using the BasicsCard and other methods that can be used to spend income managed funds and whether they had any problems
• perceived impact of income management on health of children, their own health and family relationships
• self reported impact of income management on the extent to which they experience harassment of financial exploitation
• experience of money management and financial counselling and whether the respondent thinks this helped
• housing stability and adequacy
• whether the respondent was granted a matched savings scheme payment and, if so, what the money was spent on and
• issues to do with stigma associated with being income managed.

F5.1 In-depth interviews/focus groups with Indigenous community leaders

In order to assess the impacts of the NIM on Indigenous people it is proposed to conduct in-depth interviews and/or focus groups with community leaders to discuss the implementation of income management on the community. Community Leaders will be asked about the impact of income management on children's health and wellbeing, alcohol consumption, gambling and substance misuse, and violence in the community.

F5.2 Quantitative survey of merchants accepting BasicsCards

In order to understand the impact of store licensing and income management on stores, income managed customers and the broader community it is proposed to conduct a quantitative study of merchants who are registered with Centrelink for income management.19 This survey would need to be conducted in several waves with the first during the second half of 2010 or the first half of 2011 and second wave as late as possible in the period over which the evaluation of NIM is being conducted. A small number of qualitative interviews could be conducted with merchants.

The information from the quantitative survey of merchants would be combined with data on the level of sales and the nature of goods purchased from licensed merchants and how this had changed over time. The survey could include questions about the impact of NIM on their store, what is bought using income managed money and whether income management has changed this, whether income management creates stigma, whether the BasicsCard works effectively, whether other payment methods work effectively, other effects of NIM on the community, customers understanding of income management, impact of income management on the store, and whether income management has increased their costs.

The quantitative survey of merchants could be conducted by telephone (with an option for online completion) perhaps combined with a follow-up mail out survey depending upon the type of information being sought. A further expansion would be to cover the possibility of getting scanner data from some of the major retailers.

F5.3  Survey of Centrelink staff

The principal aim of this component of the evaluation would be to provide information on: (a) implementation issues; (b) how the exemptions system works and how they assess whether an exemption to being income managed should be given and their perceptions of how well this works; (c) the interaction between Centrelink and those providing money management and financial counselling; and (d) the perceptions of staff concerning the impact that NIM is having on those being income managed and their children and dependents. A further component would be a more specific survey of the staff involved in the debriefing interviews with those coming off the old NTER scheme of income management and considering whether or not to go onto VIM, as it would appear that this would be the best way of getting data about the motivations of the groups opting into or out of VIM.

This survey would need to be conducted in several waves with the first during the first half of 2011 and the second wave as late as possible in the period over which the evaluation of NIM is being conducted. In addition, a small number of qualitative interviews or focus groups could be conducted with Centrelink staff.

It is recommended that this survey be conducted on-line given that Centrelink staff will have access to an internet connection, it allows for the survey to be completed anonymously and is the most cost-effective means of collecting quantitative data from Centrelink staff.

F5.4  Survey of money management and financial counselling support services staff

The money management and financial counselling support services will be delivered by community organisations. The services provided will include crisis support, financial counselling, advocacy and budgeting and money management education.

This component would involve collecting information from the CEO or a senior manager from the organisations providing these services and staff. The interviews with the CEO or a senior manager would collect detailed information on the type of services their organization provides, whether they were able to recruit and retain suitable staff and their views about the impact of the income management scheme and how it is being implemented. The survey of staff (generally professional staff) would collect the following information:

• professional background information about the respondent, including the job they performed
• respondents' perceptions about the efficacy of the service, including:
  • efficacy of the assessment of people's needs
  • networks/working relationships with the community
  • networks/working relationships (including referrals) with other services
  • the ability of the service to respond to people's needs
• range and nature of services provided by their organisation, including the duration and content of money management courses
• geographic scope of the services provided by their organization and
• whether there are any unintended consequences.

F6 Existing survey sources

F6.1  The National Aboriginal and Torres Strait Islander Social Survey (NATSISS)

The NATSISS was conducted in 2002 and 2008 (from August 2008 to April 2009) and the next wave is scheduled to be collected in 2013 with data not becoming available until 2014.

F6.2  Footprints in Time: the Longitudinal Study of Indigenous Children (LSIC)

LSIC aims to improve the understanding of, and policy response to, the diverse circumstances faced by Aboriginal and Torres Strait Islander children, their families and communities.

The study collects information about the lives of Aboriginal and Torres Strait Islander children, covering areas including:

• the children—their physical and mental health, how they develop socially and cognitively, their place in their family and community, and significant events in their life
• the children's families—their health, work, lifestyle, and family and community connectedness
• the children's communities—facilities, services, and social and community issues, and
• services—child care, education, health and other services used by the child and the child's family.

The first wave of LSIC, conducted in 2008, collected information from the parents and carers of two groups of children, mostly aged between 6 to 18 months and 3½ - 4½ years. A second wave was conducted between March and December 2009 and the third wave of data is being collected in 2010.

In the first wave of the Longitudinal Study of Indigenous Children, parents affected by income management/quarantining through Centrelink were asked “How is it going?” and were given the opportunity for an open-ended response. Wave 2 included an identical question. In Wave 3 study participants are being asked whether they are being income managed, whether they think income management/quarantining is a good or bad thing for their family, whether since income management/quarantining they have noticed any changes (either positive or negative) in their community and, if so, the nature of the changes they have noticed. LSIC also provides some information on the family's financial situation and how they manage their money.

The survey collected information about the lives of Aboriginal and Torres Strait Islander children in areas such as children's health, learning and development, family and community relationships, culture and identity, and community issues and services. For Wave 1, the Footprints in Time team interviewed parents and carers of 1,687 Indigenous children aged between six months and five years, from 11 sites around Australia; from Adelaide to Broome, Dubbo to Darwin, Nowra to Thursday Island.20

11. For example, if the NIM is expected to have an impact on, say, the level of alcohol consumption, while this may be difficult to measure at the individual level it would be expected that any effect should be apparent in the relative levels (or changes in the relative level) of consumption of a community where a large number of people are subject to the NIM compared areas where relatively few are impacted on by the program.
12. Data available from the Northern Territory Growth Assessment and Action (GAA) report.
13. Student Enrolment and Attendance Collection 8 November - 2008, 2009 & 2010
14. Wholesale Alcohol Supply 2001 - 2008(link is external)(Opens in a new tab/window)
15.  Dependent on whether there is any NT data on tobacco sales & if this available over time and by location
16. The Northern Territory Department of Justice publishes quarterly data on the incidence of selected crimes (Murder; Attempted Murder; Manslaughter; Robbery; Assault; Sexual Assault; Other Offences Against the Person; House Break-ins; Break-ins to Commercial or Other Premises; Motor Vehicle Theft and Related Offences; Other Theft; Property Damage; Other Property Offences) <Research and Statistics Unit(link is external)(Opens in a new tab/window)>
17. Ideally a longitudinal study which collects information shortly after commencing income management and then approximately 18-24 months later would be conducted as part of the evaluation. However, the budget for the evaluation is likely to not be sufficient for this and so it is proposed that two cross-sectional surveys of clients of income management be conducted.
18. Under the NIM a person being income managed will be able to use their income managed funds to purchase priority goods and services at stores that are not registered for income management of the BasicsCard by organising for Centrelink's Income Management Payment and Contact team to pay the store by cheque or electronically.
19. The final study data included information from:
    • 1,687 parents or primary carers of an Indigenous child.
    • 960 Study children were in the Baby cohort, approximately 6–18 months old
    • 727 were in the Child cohort, approximately 3 years 6 months to 4 years 6 months old
    • approximately 1,500 Study child questionnaires, which included information on language, height and weight
    • approximately 240 interviews with the secondary carer (Parent 2) and
    • approximately 45 questionnaires collected from teachers, child care centres or family day care providers.