Can we measure the impact of scholarships?

In 2008 the Commission sent 6,000 alumni a questionnaire about the effects of the scholarship on their personal and professional career and on social and economic developments in their country. Since then the Commission has published six reports which contain the findings of the survey. Overall, the findings are very positive: the majority of alumni return home, are able to put their knowledge and skills to good use and claim to have contributed to social and economic development in their country. These findings are very similar to those found in tracer studies of other scholarship schemes. These results should make everybody happy: the alumni, their employers, the funding agency, the education and training institutions and the agency that administers the programme.
 

Important questions

But still, uncomfortable questions keep lingering: how reliable are these findings? Do the small group of alumni who participate in a tracer study represent all alumni? Is this group biased and too positive in its responses? Is it true that most scholarship holders belong to the elite who already have better opportunities in life? Should the funding agency be satisfied with the finding that the alumni are satisfied and apply what they have learned? Should the bar be raised much higher?

As you may have guessed, the seminar was based on the conviction that more research is needed to get a better picture of the impact of scholarships and to answer DFID’s (and other donors’) most pressing question: do scholarships give value for money and is there a good return on investment?
 

How to measure impact?

The participants in the seminar consisted of a good mix of representatives of funding agencies, programme administration bodies, of researchers and scholarship holders. They tackled the question of how to define and measure the impact of scholarships and agreed that the biggest challenge is measuring impact.The results of a scholarship can manifest itself at the level of the direct benefits for the alumni (outputs), the beneficial effects of the application of the alumni's knowledge and skills on their employing organisations and institutions (outcomes), and the longer term changes in society (impact). The participants agreed that tracer studies are a good tool to cover the first two levels of results (outputs and outcomes). But impact really is about longer terms changes at a more collective level and cannot be measured through tracer studies alone.
 

More sophisticated tools needed

More sophisticated research methods need to be used and a number of methodological problems need to be tackled. One of the problems is that objectives of programmes (also scholarships programmes) may change over time. This is an important factor to consider because impact is directly related to the stated objectives of a programme. Another problem is that of attribution. There are many factors which influence a changes in our lives or in society. The more years have passed since a scholarship the more difficult it is be to attribute certain changes to it.
 

The what-if approach

The option of gathering counterfactual evidence was discussed. This is a common approach in the medical field when new medicines are tested. Counterfactuals will tell you what might have happened to a person or in a situation if no intervention had taken place (no medicine had been administered). But can such an approach be used for measuring the impact of scholarships? From a methodological perspective this would be a very challenging exercise. And a complex one as well because changes take place at different levels (the individual, an organization and society). At which level is it feasible to apply such an approach?
 

Value for money

The same goes for trying to establish the cost effectiveness of scholarships. Is it possible to establish the return on investment rate of scholarships compared to other types of interventions such as setting up a education course in a developing country? It may be possible to calculate and compare the economic benefits of two interventions, but what about the less tangible benefits for individuals or society? Scholarships provide individuals with knowledge and skills, but also expose them to new situations and cultures. They may change attitudes and introduce people to valuable networks. Can we calculate the effects of these opportunities? And if so, how?
 

The ultimate question

The participants in the seminar were unable to answer these questions but agreed that it is a challenging agenda for further exploration. When discussing these complex issues another question came up: how much time and effort should we spend on assessing the impact of scholarships?  Scholarships constitute only a small part of the overall development aid budget. Might a costly impact evaluation exercise not be out of proportion in view of the programme budgets and the positive outcomes of most tracer studies?