Appendix E: Literature review: an international literature review addressing research designs and outcomes in income management research

E1  Summary

In recent years many governments around the world have introduced conditional welfare programs, targeting particular groups within society. The philosophical basis and justification lies in the welfare contract between a society that provides financial and other support to those people who need it, and the recipients themselves who must meet certain obligations and responsibilities in return for that support from the state. The new approach to welfare rights and responsibilities is not without critics, but many commentators who have written about conditionality conclude that it can be philosophically and morally justified provided that considerable care is taken to avoid burdening those people who are already unjustly disadvantaged.

The purpose of this literature review is to prepare the ground for developing a robust mixed methods evaluation of the new income management model in the NT of Australia. Any evaluation will need to be designed to be able to monitor the progress of the new income management model and be able to provide a robust overall assessment. The evaluation should also be designed to ensure that less tangible goals such as child, family and community wellbeing are captured in the evaluation. Here we review the international literature addressing research designs and outcomes in income management research.

E2 Conditional welfare and income management

E2.1 Introduction to conditional welfare

Income management is part of a larger category of welfare programs which are generally referred to as 'conditional welfare'.  The basis of conditional welfare is to use the benefit system or cash transfers as a way of changing behaviour. Internationally the majority of conditional welfare policies and programs have been targeted at the most disadvantaged members of society and are aimed at encouraging families to send children to school and ensure they are vaccinated.  In the broadest sense most government transfers are conditional in that payment is dependent on some conditions (e.g. seeking a job, undergoing a medical examination) being met by the benefit recipient.  However in the narrower sense discussed here, conditional welfare refers to policies which use the benefit system directly to influence behaviour.  Internationally the majority of conditional welfare programs involve 'Conditional Cash Transfers'  (CCT), in which welfare recipients are given a supplement when they behave in certain prescribed ways.

In recent years many governments around the world have introduced conditional welfare programs to influence a wide range of practices within specified populations. Broadly speaking, program components are either 'punitive', with negative consequences attached to undesirable behaviour and spending on 'excluded' items such as tobacco, alcohol, gambling, pornography; or program components are 'rehabilitative' which sees desirable behaviour incentivised and 'priority' spending, on healthy nutritious food, education, housing, clothing, transport and health, rewarded. Conditional welfare programs have been widely adopted in many countries as a new approach in social assistance for combating poverty and fostering social inclusion (de la Brière and Rawlings 2006).

Conditional welfare programs differ from standard welfare programs in that they require certain income or expenditure behaviour, or a change in that behaviour, before participants receive benefits (or in order for recipients to continue to receive benefits). Changes may include increasing or decreasing expenditure, or demonstrating other behavioural change. Conditional welfare programs tend to target families with children, particularly in low and middle income families. The focus is usually on managing expenditure on behalf of children. In practice, the position of the government usually appears as though it knows best about how money should be spent on children, and certainly better than many parents within the program. Government views are usually expressed through a set of formal conditions operating within the program. Parental views and preferences are often captured using social surveys, often expenditure survey, and other forms of evaluation and reporting undertaken by researchers and public service workers who are in regular contact with families and children in the program. Populations selected as the subjects of conditional welfare programs are almost always those considered 'vulnerable', either because they are poor (low socio-economic position) or because of some other form of social disadvantage. Specific sub-groups of the population are sometimes selected as subjects of conditional welfare programs. Examples include single parent families, recent migrants, and people living in particular geographical regions or within particular jurisdictions.

The conditional welfare philosophical framework views the welfare contract in terms of a society that provides financial and other support to those people who need it and the recipients themselves who must meet certain obligations and responsibilities in return for that support from the state (Dwyer 2000). There are also mutual obligations between citizens irrespective of claims on the state. However, some critics suggest that conditionality is contrary to liberal principles; while others argue that it burdens those who are already unjustly disadvantaged (Elizabeth and Larner 2009). In many modern high income societies such as Australia and the UK conditionality has already been firmly embedded in the welfare contract between the citizen and the state. Social security and unemployment benefits, for example, are usually paid only on condition that welfare recipients are actively seeking to enter the workforce (ACOSS 2005). Many who have written about conditionality, whether in its favour or not, conclude that conditionality can be philosophically and morally justified provided that considerable care is taken to avoid burdening those people who are already unjustly disadvantaged (Deacon 2004).

2.2  A note on conditional cash transfers (CCTs) programs

As discussed above, cash transfers programs can be targeted (unconditionally) at specific vulnerable groups, or distributed universally. Another form of cash transfer program has an element of conditionality. Conditional cash transfers (CCTs) programs link the provision of cash or grants to the behaviour of the target population, who are required to perform certain verifiable actions such as securing minimum investments in children's education and health. Payment is therefore dependent on regular school attendance, or use of preventive health services or some other specified condition. Pioneered in Latin America by Brazil and Mexico during the 1990s, CCTs have spread across Latin America and many other countries around the world. CCTs are seen as a promising solution to some of the toughest challenges facing social policy. In low and middle income settings, investment in CTTs is seen particularly important in relation to meeting the Millennium Development Goals (MDGs) of reducing hunger and tackling extreme poverty and vulnerability (DFID 2006). Efforts to help poor families to invest in children's health and education will ultimately help prevent transmission of poverty from one generation to the next. Improved nutrition in young children will, in turn, benefit their health and education, lead to healthier children, and these benefits will be passed on to the next generation.

E3 Purpose of the review

The purpose of this international literature review is to prepare the ground for developing a robust mixed methods evaluation of the new model of income management in the NT of Australia.

The evaluation methodology must be designed to be able to assess and monitor the progress or 'health' of the new income management measure in the Northern Territory. FaHCSIA has also made it clear that the evaluation should be designed to ensure that less tangible goals such as child, family and community wellbeing are captured in the evaluation. Here we review the international literature addressing research designs and outcomes in income management research.

E3.1  A note on the literature review

The international review of literature was exploratory rather than systematic and confined to the English-language literature. The starting point was the authors' existing knowledge of the literature and included searches in Google Scholar, followed by more specialised databases such as IngentaConnect, Project Muse and Cambridge Journals Online and the Medline to cover the health journals. Other sources were identified and included as their relevance became clear during the research process. Key words for the search were: income management, conditional cash transfers, conditionality, social protection, welfare, family income and expenditure. The key inclusion criterion for the literature review was the relevance to understanding research designs and outcomes in income management research.

E4 Conditional welfare programs

There are a number of conditional welfare programs. These are described below.

1. Opportunity NYC–Family Rewards (USA)
   In 2007, New York City's Center for Economic Opportunity launched Opportunity NYC–Family Rewards, an experimental, privately funded CCT program to help families break the cycle of poverty. Family Rewards is the first comprehensive CCT program in a developed country. Aimed at low-income families in six of New York City's highest-poverty communities, Family Rewards ties cash rewards to pre-specified activities and outcomes in children's education, families' preventive health care, and parents' employment. The three-year program is being operated by Seedco (a private, non-profit intermediary organization) in partnership with six community-based organizations. The program is being evaluated by MDRC (a non-profit, nonpartisan social policy research organization) through a randomised control trial involving approximately 4,800 families and 11,000 children, half of whom can receive the cash incentives if they meet the required conditions, and half who have been assigned to a control group that cannot receive the incentives.
   Despite initial challenges in understanding the program's large number of incentives and related payment requirements, nearly all families eventually earned rewards, more than $6,000, on average, over the first two years (Riccio, Dechausay et al. 2010). In addition, the program resulted in a number of other outcome measures:
   • evidence of reduced poverty and hardship, including reduced hunger, housing and health care hardships
   • increased savings and the likelihood that parents would have bank accounts
   • increased school attendance, but did not improve school outcomes or grades overall
   • increased use of health insurance coverage and use of health services and dental care services, and
   • increased employment levels.
2. Food Stamp Program (USA)
   The United States Supplemental Nutrition Assistance Program (SNAP), historically and commonly known as the Food Stamp Program, is a federal-assistance program that provides assistance to low- and no-income families. Though the program is administered by the U.S. Department of Agriculture, benefits are distributed by individual States. The program provides families with stamps that can be used to purchase eligible foods from eligible retailers. Stamps are provided to families on a means-tested basis, and are supplemental to other forms of welfare, rather than replacements. During the 1990s the program was revamped and actual stamps were phased out in favour of a specialised debit-card system, the Electronic Benefit Transfer. Evaluation of the program suggests that food stamps help to reduce poverty in part by freeing up other income without reducing the quality of food available in the household (Jolliffe, Gundersen et al. 2003).
3. Temporary Assistance to Needy Families programs (USA)
   The Temporary Assistance to Needy Families (TANF) program covers a range of programs in the United States which differ between States. In some States, individual Counties (similar to Australia's Local Government tier) may implement customised programs. The main aim is to provide temporary assistance for low income families with children under the age of 18. Programs include income support while looking for work, special payments and intervention programs for children and work placement schemes. Access to programs is usually dependent on child enrolment in school and adults actively seeking employment. Different evaluations have been carried out in different jurisdictions; however the general consensus is that the program has some positive effects on employment rates, but impacts on the lived experience of poverty and family incomes remain less clear (Moffitt 2003).
4. Self Sufficiency Project (Canada)
   The Self Sufficiency Project (SSP) in Canada was conceived and funded by Human Resources Development Canada. The project offered a temporary earnings supplement to selected single-parent families receiving Income Assistance (welfare) in British Columbia and New Brunswick. Between November 1992 and March 1995, more than 6,000 single parents who were long-term Income Assistance (welfare) recipients were invited to join the SSP research study. To collect the supplement (a monthly cash payment based on actual earnings), a single parent had to work full-time and leave Income Assistance. They were then eligible to receive the supplement for up to three years, as long as the single parent continued to work full-time and remained off Income Assistance. The supplement roughly doubled the earnings of many low-wage workers (before taxes and work-related expenses). The Self Sufficiency Project found positive outcomes for employment for adults and school performance for children. Poverty reduction impacts only persisted for the time earnings supplements were provided, but child education impacts persisted (Michaelopoulos, Robins et al. 2005).
5. Progresa (now Oportunidades) program (Mexico)
   In Mexico, the seminal Progresa program (now Oportunidades) began in 1997 and aimed to improve the health and education outcomes of low-income children. Households, selected on socioeconomic criteria, were given special cash payments provided that children regularly attended both school and appointments for preventive health care. Participating children aged 4 to 23 months were also given food supplements. Nutrition: research suggests that 70 percent of households participating in Progresa in Mexico have shown improved nutritional status. There have been reductions in stunting and the growth rate among infants has increased (Skoufias and McClafferty 2000; IFPRI 2002). Health: research on the Progresa program operating in Mexico suggests that the scheme boosted demand among women for antenatal care by 8 percent, which contributed to a 25 percent drop in the incidence of illness in newborns and 12 percent lower incidence of ill-health among children compared with non-Progresa children (Skoufias and McClafferty 2000). Immunization against measles increased by a mean of 3 percentage points 6 months after the beginning of the Progresa program and tuberculosis vaccination was 5 percentage points greater for infants than at baseline. However, there was less evidence of improvement in school outcomes or post-school employment opportunities.
6. Red de Proteccio´n Social (Nicaragua)
   In Nicaragua, the Red de Proteccio´n Social program was designed in a similar manner to the program in Mexico. Disadvantaged households in low-income areas received a cash transfer provided they brought their children who were younger than 5 years to preventive health examinations (where they received antiparasitic drugs, vitamins, iron supplements and vaccinations) and attended health education workshops. An additional cash transfer was contingent on enrolment and regular attendance at primary school. In Nicaragua's Red de Protección Social program, immunisation levels among recipient infants increased by 18 percent. Vouchers provided to sex workers and their clients for the treatment of sexually transmitted infections led to large declines in reported rates of syphilis (9%) and gonorrhoea (5.25%) (Sandiford, Gorter et al. 2002; Ensor 2004).
7. Familias en Acción (Colombia)
   In Colombia, the program Familias en Acción targetted the poorest households in disadvantaged municipalities. The program provided monetary transfers to mothers on the condition that their children who were younger than aged 7 years attended preventive health examinations, and another transfer if their children aged 7 to 17 years attended school regularly. Mothers were also encouraged to attend health education courses. Evidence from the Colombian program suggests an increase in children's preventive health care visits by 23 percentage points for children younger than aged 2 years and 33 percentage points for children aged 2 to 4 years (Attanasio, Battistin et al. 2005;
8. Programa de Asignación Familiar (Honduras)
   In Honduras, any household in municipalities with high prevalence of malnutrition and benefiting from the Programa de Asignación Familiar had access to two types of monetary incentives: one conditional on school attendance of children aged 6 to 12 years, and the other conditional on undergoing monthly health examinations for children and prenatal care attendance for pregnant women (Morris, Flores et al. 2004). The evaluation of the Honduras program Programa de Asignación Familiar showed a mean increase of 6.9 percentage points in the coverage of the first dose of diphtheria, tetanus, pertussis vaccine among children younger than 3 years.
9. Bolsa Escola (now Bolsa Familia) (Brazil)
   In Brazil, Bolsa Escola (now Bolsa Familia) is a national conditional welfare program that transfers $6-19 a month to an estimated 5 million poor families, at a cost of 0.15 percent of gross domestic product (Rawlings 2004). The program aims to improve child and maternal health among low income populations. Mothers received capped transfers based on the number of beneficiaries (either children younger than aged 7 years or pregnant or lactating woman) in the household. Transfers are conditional on attendance at health check-ups and nutrition workshops for the women and adherence to vaccination schedules for children. Education Brazil's program makes cash transfers to mothers with school-age children on the condition that school attendance rates of 90% are achieved. Research shows falls in school drop-out rates and higher enrolment rates in education (UNESCO 2003).
10. Mi Familia Progresa (My Family is Moving forward) (Guatemala)
    In Guatemala, the program Mi Familia Progresa (My Family is Moving forward) was introduced relatively recently in 2008. This program provides cash payments to poor mothers, conditional upon them sending their children to school and for health check-ups. Two types of monetary transfers, both targeted to women. A monthly health and nutrition transfer of around US$ 18 (Quetzales 150) is paid to mothers of children under the age of six, to pregnant women and to breast-feeding mothers, under the condition that they attend health centres to receive a basic package of nutritional and preventive maternal-child health care services. An education transfer of US$ 18 (Q 150) is paid to families with at least one child between the ages of 6 and 15 attending primary school or preschool. If more than one child in the family is within the prescribed age, all of them must show proof of school attendance for the mother to receive the monthly transfer. The two types of transfer are not mutually exclusive and they can be cumulated provided that the family qualifies for both and complies with the conditions. The amount provided is not adjusted to the number of children, size of the family or other special conditions, probably to avoid perverse fertility incentives but also to maximise the number of beneficiaries across the country. Given the short life of Mi Familia Progresa, there is little in the way of preliminary results to-date (Gaia 2010).

11. National Family Welfare Program (India)
    India launched the National Family Welfare Program in 1951 with the objective of reducing the birth rate to the extent necessary to stabilise the population at a level consistent with the requirement of the national economy. The program provides incentives for women to spend money on health related activities for themselves and their children. There is a strong focus on family planning including subsidising contraception and reproductive health. The program has been linked to reduced rates of fertility and infant mortality (World Bank 1995), for example, the achievements of the program at the end of the 7th five year plan (1985-90) were:

    • reduction in crude birth rate from 41.7 (1951-61) to 30.2 (1990)
    • reduction in total fertility rate from 5.97 (1950-55) to 3.8 (1990)
    • reduction in infant mortality rate from 146 (1970-71) to 80 (1990)
    • increase in couple protection rate from 10.4% (1970-71) to 43.3% (1990)
    • over 118 million births were averted by the end of march, 1990.

    Disentangling the distinct impacts of the project from other impacts such as increasing education and living standards, and increasing employment rates among women, remains an issue, particularly given the length of time the program has been running (for over 50 years). The National Family Welfare program continues to evolve and continues to target child and maternal health issues.

12. The Female Secondary School Stipend Project (Bangladesh)
    The Female Secondary School Stipend Project in Bangladesh was established to increase the enrolment of girls in secondary schools, thereby delaying marriage and childbearing. The program paid school fees and transferred an incentive payment direct into girls' bank accounts on condition of at least 85% school attendance. Recorded enrolments in program schools increased by 105% between 1994 and 1999 while aggregate enrolments of girls increased by 111%. In rural areas, enrolment rates increased by 12 percentage points per year. Unfortunately, accurate data to measure the specific impact of the program on enrolment do not exist. The School Stipend was one among many other interventions pushing for increased enrolment of girls, including increased focus on non-formal education and tuition waivers, and it is difficult to extract the impact of the stipend from these other causal factors (Raynor and Wesson 2006; Schurmann 2009).
13. The Zomba Cash Transfer Program (Malawi)
    The Zomba Cash Transfer Program (ZCTP) in Malawi was a randomised conditional cash transfer intervention program that provided incentives – in the form of school fees and cash transfers – to schoolgirls and young women who have recently dropped out of school to stay in or return to school. Baseline surveys were conducted with 3,805 girls and young women between October 2007 and January 2008. The study reported that the percentage of initial dropouts who returned to school and were in school at the end of the 2008 school year was 17.2% among the control group compared with 61.4% among the treatment group. Thus, program beneficiaries were 3-4 times more likely to be in school at the end of the 2008 school year than the control group. After one year, the program also reported declines in early marriage, teenage pregnancy, sexual activity, and risky sexual behaviour (Baird, Chirwa et al. 2009).

E5 Assessment of conditional welfare programs

E5.1 Impact assessment

Overall, the evidence suggests that conditional welfare programs are effective in increasing the use of health and education services, which essentially means they have had demonstrable impacts on the health and education related behaviours in the target populations. There is also good evidence that conditional welfare programs can improve nutrition and health status among target populations. Many issues remain however and conditional welfare programs continue to provoke much debate within social policy. As conditional welfare programs are a relatively recent contribution to social policy, longer-term impacts are less clear. The cost-effectiveness of many conditional welfare programs is often uncertain, particularly when compared with supply-side interventions such as improving infrastructure and services. While evidence indicates that even when cash transfers are not tied to service use, the additional income is often used for health, education and nutrition priorities. A recent evaluation of the impact of an emergency unconditional cash transfer scheme in Somalia, for example, showed that the provision of cash grants to women not only helped the poor repay debts and improve their food intake, but also empowered them to invest in their own health care (Ali, Fanta et al. 2005). Unconditional cash transfers are usually seen as being more appropriate where service coverage is poor.

There are also structural issues to consider. Social transfers are not a panacea to compensate for a lack of investment in equitable education and health services. Conditional welfare programs, to stand any chance of success, will need to operate in those areas where there are adequate services, schools, clinics and the like. In settings where access to public services is low, expanding capacity may be a preliminary step before the introduction of conditional welfare programs. User fees for health and education services may need to be abolished, waivered or reduced. Fee waivers are often an important complement to social transfers, provided adequate measures are put in place to meet the costs. This will likely increase the impact of both the social transfer and the fee waiver or free service policy although further research is needed on the combination of social transfers and supply side interventions that are most effective in different settings.

There may be issues relating to the size of the transfers needed to change behaviour in different settings; there may even be inefficiencies in conditional welfare programs. A program may yield quite a high marginal cost per outcome, particularly if money goes to all individuals in a targeted population regardless of compliance with the conditionality of a program. Ideally, positive outcomes of conditional welfare programs should be weighed against their cost effectiveness, particularly when conducting robust evaluations, but cost effectiveness is an area which is underdeveloped in this field as Lagarde et al. (2007) observe.

E5.2  Issues in the design and evaluation of conditional welfare programs

Decisions about the type and value of social transfers are extremely context-specific. Very often, the choice and design of social transfer programs reflects the priorities and political realities of policymakers, as much as technical feasibility, institutional capacity and affordability. There are a number of issues to consider in the design and evaluation of conditional welfare programs:

E5.3  Policy objectives

Is improving human development an explicit policy objective or an indirect objective of a program to reduce and poverty? When deciding which transfer is more appropriate in certain conditions, it is important to keep in mind that governments have multiple objectives. If a government wants, for example, to provide income support to tackle poverty and improve human development outcomes, then both conditional and unconditional transfers are likely to be the most appropriate choice. There may be broader social policy considerations to think about. There may be merit, for example, in combining transfers with other policy options, such as the possible removal of a service user fee. The conditional welfare program can provide a platform for coordinating overall social protection strategy, social policies and sector strategies that together contribute to improved education and health goals.

E5.4  Understanding demand

What types of financial and non-financial barriers to access need to be addressed? What is the optimal value of transfer required to overcome barriers for vulnerable groups? Who in the household should receive transfers to maximise impact? In some contexts it may be a good idea to conduct pilot work, before rolling out the conditional welfare scheme.

E5.6  Cash or restricted spending choice

Is there adequate provision of quality services to tie transfers to service use? Do vulnerable people have sufficient information to help make informed choices, if not additional efforts may be required to ensure information needs are met. To what extent are there political pressures or impediments to restricting spending choices through conditions or vouchers; is there the capacity to monitor compliance with conditions? Income management programs could also be complemented with a fee waiver scheme or, where appropriate, entitlement to free service provision. The transaction costs of alternative policy options will need to be considered and estimated. Vouchers work tend to be more successful when targeted to easily defined vulnerable groups and they often work best when the main cost involved is paying for the service, and there are no additional out of pocket expenses involved, or people are not giving up work time to access services.

E6. Targeted and universal approaches

What are the trade-offs between accuracy of targeting and the politics and cost of targeting? Are there existing programs that use effective approaches to targeting, e.g. geographical, community-based or proxy means-testing scheme that a social transfer scheme can also use?

E6.1 Cost-effectiveness

Will the benefits of social transfers – in terms of human development outcomes – exceed the costs of administering them?

E6.2 Political feasibility

What is the constituency for social transfer programs? Will financing be sufficiently predictable and long-term to sustain social transfer programs?

E6.3 Capacity, governance and accountability

What are the options for delivering cash and vouchers directly to people (e.g.  using banks, post offices or health centres); which different institutions and levels of government administration need to cooperate to implement social transfers effectively and in coordination with education and health service provision?

E6.4 Evaluation

Usually there is an imperative to monitor and evaluate social transfer schemes. Many cash management programs have been the subject of strict randomised controlled trials which help to ensure that the income management program effects are captured (see the review of evaluation methods, below). There may be opportunities to collaborate with other agencies on rigorous evaluations of social transfer programs to ensure that human development and poverty reduction objectives are addressed in order to build the evidence base.

E7  Review of methodologies for income management programs

E7.1  Hierarchy of evidence

The strength of evidence provided by research is dependent on both the methodology used and the quality of the particular study. Evidence hierarchies have been developed to help researchers to rank different research methods according to the validity of their findings, based on the risk of error or bias in their results. Figure 10.1 shows the conventional hierarchy of research evidence which puts randomised techniques above controlled and controlled above quasi experimental techniques which tend to rely on evaluation before and after an intervention. We have used the hierarchy of evidence to guide our thinking for this review; the hierarchy of evidence has also been used to structure our literature review of research designs and the results are summarised in Table 10.2.

A randomised control trial would be the gold standard for any aspiring new study looking to investigate any cash transfer or income management programs. The vast majority of studies reviewed in Figure 10.1 have adapted either a randomised controlled or a controlled study design (without randomisation of participants to treatment or intervention groups). Often researchers may rule out the gold standard of random allocation for political reasons or because it is unethical in community evaluations, and in this context the next best approach would be a 'stratified random' study which randomly allocated areas for introduction of the policy, or more commonly a controlled study in which matched locations or towns as similar as possible to treatment towns are identified, in order to try and identify any effects and outcomes of the income management program. A before and after quasi-experimental design would be the least desirable form of evaluation simply because it is hard to attribute change to an intervention. As shown in Figure 10.1, a systematic review of the results from randomised controlled trials and studies is usually considered the pinnacle of the research hierarchy of evidence.

Figure 10.1 Hierarchy of evidence

Figure 10.1 illustrates the hierarchy of evidence used to guide the researchers conducting the evaluation to rank different research methods according to the validity of their findings, based on the risk of error or bias in their results. Figure 10.1 shows the conventional hierarchy of research evidence which puts randomised techniques above controlled and controlled above quasi experimental techniques which tend to rely on evaluation before and after an intervention.

E7.2  Methodologies, randomisation and controls in income management trials

A good example of a randomised control study design to investigate the impact of an income management program is provided by the recent experience in Malawi as discussed above. Here, 3,821 young women living in villages in Zomba, a district in Southern Malawi, were allocated to receive the intervention (treatment), half received the intervention and the rest serve as the control group. The following schematic in Table 10.1 captures the research design. The program provided incentives – in the form of school fees and cash transfers – to schoolgirls and young women who have recently dropped out of school to stay in or return to school. Within each treatment community or village, all never-married 13-22 year-old recent dropoutswho are eligible to return to primary or secondary school were identified and were always treated (with conditional cash transfers).

Table 10.1 Malawi research design

The schematic in Table 10.1 captures the research design. The specific example used provided incentives – in the form of school fees and cash transfers – to schoolgirls and young women who have recently dropped out of school to stay in or return to school. Within each treatment community or village, all never-married 13-22 year-old recent dropoutswho are eligible to return to primary or secondary school were identified and were always treated with conditional cash transfers.

Source: Baird, Chirwa et al. (2009; 2009).

The study reported that the percentage of initial dropouts who returned to school and were in school at the end of the 2008 school year was 17.2% among the control group compared with 61.4% among the treatment group. Given the strict randomised control design, we can be fairly confident that the outcomes observed resulted from interventions in the treatment group; girls in the treatment group were 3-4 times more likely to be in school at the end of the 2008 school year compared to girls in the control group. After one year, the program also reported declines in early marriage, teenage pregnancy, sexual activity, and risky sexual behaviour (Baird, Chirwa et al. 2009).

As noted above, a randomised controlled study design is the gold standard for evaluating a new income management program. Accordingly, many of the studies reviewed in Table 10.2 have adapted such a design. Often researchers may rule out random allocation for political reasons or because it is unethical in community evaluations. In this context, the next best approach would be a matched study in which control locations or towns are chosen to be as similar as possible to treatment towns in order to try and identify any effects and outcomes of any new policy. In some instances a 'before and after' quasi- experimental design may be the only option. However, this design would be the least desirable form of evaluation simply because it is hard to attribute change to the intervention as other things that we cannot control, such as a rise in living standards or a general education campaigns, may play a role in changing people's behaviour. There may also be other factors which we are not able to observe. In a before and after design the outcomes of the intervention group are compared before and after the intervention. This type of evaluation relies on the availability of good baseline data on the variables of interest, including expenditure patterns, people's money management skills, and indicators and measures of the wellbeing of children, families and communities. However, realistic evaluation, particularly in the social sciences, must observe research principles, consider levels of funding, observe ethics, and politics, time constraints and routine datasets that may already be available to assist with any planned inquiry. In all research studies there are trade-offs to be made.

E8 Evaluation instruments used in income management program evaluations

In Table 10.2 we have also considered the range of instruments that researchers have used in their evaluations, we have summarised the results under broad categories including social surveys (which includes inquiries where questionnaires and/or personal consumption diaries are used), routine administrative data collected by schools and hospitals and other public bodies, face-to-face interviews and focus groups. It is clear from the review that researchers often make use of multiple instruments to evaluate the impact of income management, often involving mixed methods of qualitative and quantitative techniques.

Actual expenditure behaviour changes may be considered the primary outcome in the income management program. Second level outcomes relate to policy objectives that go beyond the immediate program. These may include increased school attendance or educational attainment, improved public safety, health and economic outcomes such as entry in the labour market. As mentioned above, outcomes arise from complex systems of which specific programs are one usually only component, causal relationships are often difficult to establish and usually require randomised controlled study designs. There is good evidence, as we saw above, to support a link between conditional welfare programs and increased school attendance of children, better use of health services and evidence of adults returning to work. In any evaluation great care needs to be taken to avoid overburdening the families involved in the programs that are likely to be disadvantaged already.

[ top ]

This table shows a review of methodologies and evaluation instruments of international conditional cash transfer schemes. Each study and its origins are assessed based on the nature of the evaluation design and its study group analysis. Evaluation instruments highlight the different methodologies and information sources for each individual study of conditional cash transfer schemes.

*Indicates use of MIS IDA Data – a software program developed by the Center for Social Development for use in the management of IDA programs. The program assists organisations in managing IDA accounts, program administration and evaluating programs (Carpenter, 2008: 5)