NALA, the National Adult Literacy Agency, published a report in July entitled ‘Literacy Impact: An Outcomes Framework for measuring the impact of improved literacy, numeracy and digital skills’.
The report presents a proposed framework for measuring outcomes and impact on the lives of adult learners whose literacy, numeracy and digital skills are improved through education and literacy strategies and initiatives in Ireland.
The report proposes indicators that could be used to measure evidence of impact, identifies methods and tools for gathering data and evidence, and makes recommendations for how the proposed Outcomes Framework and indicators could be featured in governmental and other policy strategies.
In the introductory section, key words are discussed with “indicator” is defined as:
An Indicator is the specific measure of target used to identify whether an outcome has been achieve, or partially achieved. Indicators should be specific and measureable.
Examples of indicators given include achievement of qualifications, demonstration of new skills, and then learner expressing increased confidence about new skills. The report also states that indicators can be “hard” or “soft”. Examples of hard indicators include attendance recording and using qualifications data to see qualifications achievements. Soft indicators could be learner statements, feedback to surveys/research, or focus groups.
The report, written by Jackie Borge (Borge Consulting) and Daniel Sellers, then discusses the methodology of gathering evidence of the benefits of improved literacy, numeracy and digital skills through reviewing 96 documents and resources including NALA thematic papers as cited in the bibliography. Interviews were also carried out with six international experts.
Outcomes Framework
Through this research, the Outcomes Framework was produced by identifying categories of outcomes, domains and sample indicators of the benefits of higher literacy, numeracy and digital skills. Three broad categories of outcomes were defined:
- Educational (referring to benefits for the individual)
- Life-wide (for the individual and those around them)
- Economic and Societal (for the country or population at large)
Within these proposed three categories are then eight domains, or areas of life, where these outcomes present:
The domains we propose reflect long-recognised themes for understanding the wider benefits of adult literacy, numeracy and digital skills, and of adult learning in general.
These domains, represented as part of the above categories, and some sample outcomes are:
- Educational:
- Learning and Personal Development:
Outcomes relate to the personal acquisition of literacy, numeracy and digital skills; wider personal development including developing self-esteem and aspirations; and the learner’s development as a lifelong learner with increased capacity and capability to continue to learn.
- Life-wide:
- Family and Intergenerational: This domain focuses on outcomes that impact not only the adult/learner, but also the family around them – for example, the learner is able to help their children with homework because of their improved literacy/numeracy. Another outcome in this domain is that education is more highly valued within the family.
- Community Life and Social Inclusion: Outcomes in this domain relate to the learner’s increased engagement in their community and also deals with their experience of social inclusion.
- Health and Wellbeing: Outcomes in this domain centre on the improvement of the learner’s physical and mental health and wellbeing as a result of improved literacy, numeracy and digital skills.
- Work: Outcomes under this domain relate to improved employment prospects and performance due to improved literacy, numeracy and digital skills, as well as skills development in general. Another outcome identified is that learners have increased capacity for alternative means of income.
- Financial Capability: This domain’s outcomes often align with those from the ‘Work’ domain in that they pertain to the learner’s ability to manage employment and improved financial circumstances. However, financial capability also relates to the range of skills and knowledge such as budgeting, understand banking, understanding economic matters and knowing how to get help when faced with financial difficulty.
- Societal and Economic:
- Society and Citizenship: These outcomes relate to “social trust” and include learners’ greater societal understanding, participation and democratic engagement. Increases in civic engagement and civic responsibility are also outcomes in this domain.
- Economy: Outcomes in this domain relate to the benefits of increased efficiency and productivity, as well as the overall economic improvement as a result of higher levels of literacy, numeracy and digital skills in the population/workforce.
Measuring Outcomes
The report discusses challenges relating to measuring outcomes and that many of these outcomes are not necessarily valued by policy makers as many are evidenced mostly by learners’ feedback and experience. These kinds of outcomes are often described as “soft” outcomes, as opposed to “hard” outcomes (which would be related to work or more concretely measured by databases, etc., such as economic or financial improvements.)
Citing a report published by the National Institute of Adult Continuing Education (UK), the report comments on the importance of these “soft” outcomes when analysis the data:
It may be that other outcomes, such as increased self-confidence and self-esteem, are more significant in encouraging and enabling learners to take the next step in their learning journey.
The report then goes on to list some of the challenges and considerations in measuring and applying the data around outcomes for adult learners in order to get a full picture of benefits and needs, which include:
- Measuring outcomes where multiple partners or policies are involved. Here the report cites the Organisation for Economic Co-operation and Development, saying that data relating to the outcomes of non-formal education “rarely exists” and that “different stakeholders must join forces, reflect on what information should be gathered, how to set up the data collection process in practice, and agree on the use of data.”
- As addressed above, sometimes only data relating to improved literacy, numeracy and digital skills that is seen as “valuable” is measured and collected.
- Learner’s literacy ability when responding to surveys/assessments.
- Ensuring that practitioners are confident and skilled enough to apply measurement tools.
The report then discusses the spectrum of methods and tools for measuring outcomes:
These methods and tools span a spectrum: from ‘softer’ methods and tools such as those applied in a learning setting, to ‘harder’ methods and tools such as large-scale research studies or economic analyses.
Some examples of ‘soft’ measures are individual learning plans (ILPs), learner surveys and evaluations, and case studies of individuals’ experiences. Some examples of ‘hard’ measuring methods might seek to show the increase GDP of a country based on having a more skilled population, increased skilled workforce numbers, and reduction in welfare or health budgets.
A comprehensive set of sample indicators and outcome measurements is featured in Appendices C and D.
Recommendations
The final section of the report gives recommendations on how the Outcomes Framework could be applied to policy and strategies on literacy, numeracy and digital skills in Ireland. These include Whole-of-Government strategies; endorsing mixed methods approaches to measuring outcomes; and the recommendation to plan training for practitioners/managers in related services on how to build outcome measures into their planning.
You can read the full report here: https://www.nala.ie/wp-content/uploads/2020/07/Literacy-Impact-NALA-2020.pdf