Actor-oriented criteria
RAISD-project aims to find feasible strategies in alleviating the vulnerabilities of forcibly displaced people in most deprived positions. To accomplish this task, a solid evaluation methodology is needed. In short, evaluation means defining the value of a certain program or an intervention.
The foundation of the evaluation methodology are criteria that have been set by local stakeholders in each partner country. These actor-oriented evaluation criteria have been carefully analysed and merged as an integrated evaluation criteria serving as a basis for evaluating the Tailored Attention and Inclusion Strategies developed (TAIS) in RAISD. The integrated evaluation criteria provide a framework which need to be adjusted to fit each context and the singularity of each TAIS.
In theoretical terms, the integrated criteria are congruent with the principles of realist tradition of evaluation. In addition to results, realist evaluator is interested in how and why the results are (or are not) obtained. The same strategy might work for some people in some contexts but fail in different surroundings. The aim is thus to analyse the TAISs as both processes and outcomes from a multi-stakeholder perspective.
From a more practical perspective, TAISs as processes are evaluated by studying how accessible it is, to what extent people use it, what kind of resources it provides for different stakeholders and how easy it is to set up. Further, evaluating the TAISs as outcomes pushes one to look for changes in the level of capabilities and competences of the involved stakeholders.
Framework for Integrated evaluation criteria
PROCESS EVALUATION
The level of ease TAIS is reached by beneficiaries
- Statistics: Number of beneficiaries & distributions of them (according to gender, ethnicity, ability etc.)
- Qualitative feedback from beneficiaries: Ease of access/use & non-discrimination
- Qualitative feedback from stakeholders: Potential beneficiaries not reached (and why it was so)
How TAIS is perceived by the beneficiaries?
- Statistics: Adherence (e.g. intensity of use or level of attendance) and withdrawal rates (if possible, according to gender, ethnicity, ability etc.)
- Qualitative feedback from beneficiaries or stakeholders: How TAIS match needs?
- Qualitative feedback from beneficiaries or stakeholders & observations: How TAIS is received?
What types of resources TAIS provides or what types of changes it provokes for the beneficiaries?
- Qualitative feedback & observations: How pilots are used and for what purposes?
- Might be different than originally planned
- Qualitative feedback & observations: What kind of resources and/or recognition TAIS provides for the beneficiaries?
- Might be different than originally planned
Feasibility of the TAIS from the service provider perspective
- Statistics: Frequency of using/providing TAIS
- Statistics: Increased cross-professional contacts as a result of TAIS
- Qualitative feedback: Ease of providing TAIS
- Qualitative feedback: How TAIS match with professional/institutional culture?
OUTCOME EVALUATION
To what extent TAIS influences beneficiaries’ perceived autonomy and coping in the context of host communities?
- Follow-up surveys and/or retrospective interviews: Change in the level of autonomy (e.g. perceived ability to cope in one’s surroundings and belief in one’s abilities in the future)
- Follow-up surveys and/or retrospective interviews: Change in the level of perceived coping skills (e.g. communicating and awareness on one’s possibilities)
To what extent TAIS influences societal participation of beneficiaries?
- Follow-up surveys and/or retrospective interviews: Change in social networks & community engagement
- Follow-up surveys and/or retrospective interviews: Change in employment or educational status and change in the level of access to services and institutions
To what extent TAIS influences competences/skills of beneficiaries?
- Follow-up surveys and/or retrospective interviews: Knowledge on work and educational opportunities, and on the rights and available services
- Follow-up surveys and/or retrospective interviews and/or tests: Change in language skills and societal/cultural knowledge
To what extent TAIS influences competences/skills of service providers?
- Retrospective interviews/feedback: Learning instigated by new methods of service provision or cross-professional co-operation
- Retrospective interviews/feedback: Professional/institutional transformations as a result of providing the TAIS