Evaluation approach

In keeping with the complex nature of the Childsmile initiative, its evaluation comprises a multi-faceted approach founded upon Childsmile’s programme theory.

A comprehensive evaluation strategy

The theory-based approach to evaluation uses key stakeholders’ views about how the Childsmile programme works (i.e. how programme activity contributes to outcomes) to guide the evaluation.

The result is an ‘outcome-focused’ model of evaluation. However, evaluation methods focus not only on assessment of impact in terms of outcomes, but also on the processes involved in the implementation of the intervention. In addition, an integrated evaluation design is adopted and, where feasible, experimental trials are undertaken to test the effectiveness of component activities.

Implementation of the theory-driven approach

The Central Evaluation and Research Team (CERT) and an external consultant, Avril Blamey and Associates (ABandA), organised and facilitated two workshops to develop Childsmile's programme theory and agree an evaluation strategy based upon it. Participants included key strategic and operational staff and the programme funder.

CERT and ABandA drafted logic models for four programme elements and charted their underlying assumptions prior to the first event. Logic models, in this context, are a representation of what a programme is trying to achieve and demonstrate the links between intended inputs, activities, outputs and outcomes. Assumptions are statements describing potential weaknesses in the underlying logic, i.e. the programme theory, of these models.

Workshop 1: Stakeholders assessed the models for plausibility, do-ability, and testability and reviewed three overarching programme assumptions which stated that: planned targeting and tailoring of Childsmile activities will reduce inequalities in oral health; dental health services can be re-oriented towards prevention; and best practice from Childsmile embedded in mainstream services will lead to improved oral health. Participants were encouraged to suggest additional assumptions and sub-assumptions and to identify which constituted the largest potential risks to the programme.

Workshop 2: Stakeholders reviewed evaluation activity and data availability against the revised Integrated Logic Model and the issues raised by the assumptions exercise. This identified any ‘gaps’ in evaluation activity and participants prioritised these areas for evaluative focus. Topics thought to be important included:

  • the quality of service delivery
  • families and children’s engagement with and satisfaction with the programme
  • professional engagement with and support for the programme
  • the potential for reductions in health inequalities.

Existing evaluation work streams were reviewed and modified to ensure that all areas prioritised by stakeholders were aligned with one or more of the key domains.

This matrix below shows the resultant evaluation domains:

3.12.2_evaluationmatrix

Next steps

Studies are underway or planned to address each of the domains above. You can also view further information about ongoing research and evaluation activity in selected areas of work.

In accordance with Childsmile’s theory-based evaluation approach, the programme’s theory of change is continually reviewed in an iterative process. It is expected that the evaluation matrix will evolve with the development of the programme but key domains will remain.