Malawi Evaluation 2015

Malawi evaluation 2015: Independent Evaluation of Signal’s
Awareness and Communication Training for Deafness (ACT) Project
in Northern Malawi

Project funded by Comic Relief
Project implemented by Church of Central Africa Presbyterian (CCAP)
Evaluation conducted by Lifetime Consulting in May 2015

Link to full evaluation

Executive Summary

Background

With funding from Comic Relief, the ACT Project was implemented between 2013 and 2015 by the Church of Central Africa, Presbyterian (CCAP) with support from Signal. The ACT Project, which built on Signal’s previous work in Malawi, aimed to improve access, retention and achievement of deaf and hearing-impaired children (HIC) in mainstream primary education. It was expected that by the end of this project, deaf / HIC would be accessing inclusive, equitable and relevant education, which would enable them to become full and active members of their families, communities and societies. The geographic focus was four districts in the Northern Region of Malawi, namely Mzuzu City, Mzimba North, Nkhata Bay and Likoma Island. The key target groups are deaf and hearing-impaired children; their parents and guardians; mainstream teachers; and community representatives in these districts. Other key stakeholders included community members in the four districts; local and national representatives of the Ministry of Special Needs Education; District Education Managers (DEMs); and Primary Education Advisers (PEAs). 

Study methodology

This Malawi evaluation 2015 was conducted as a comprehensive process that included assessing relevant project documents and fieldwork activities in three out of the four districts where this project was implemented, i.e. Mzuzu City, Mzimba North and Nkhata Bay. The evaluation team, which was comprised of one team leader, a local consultant and a team of local research assistants utilised both qualitative and quantitative methodologies to draw out perspectives of deaf/hearing impaired children, teachers, heads of schools, parents, communities and other key stakeholders. Individual structured questionnaires were administered to 405 project beneficiaries from Mzimba North and Mzuzu City. Of the 405 survey participants, 21% were deaf/hearing impaired children; 26% were children without hearing impairment; 26% were teachers; and 27% parents of hearing impaired children.  Of this study population, 52% were female and the rest, i.e. 48%, were male. In addition, twelve focus group discussions were conducted with 166 participants from various categories of people highlighted above. Furthermore, up to 40 key informants were interviewed, including District Education Managers, Regional Education Managers, SNE coordinators, national stakeholders, as well as, staff representatives from CCAP, Signal and Comic Relief.

Summary of main findings

Overall, the evaluators observed that the two year “Awareness and Communications Training for Deafness” project has made significant strides to overcome barriers to equitable access to education for hearing impaired children (HIC) in Northern Malawi. The evaluation team concluded that the project was well delivered and successful, despite having been unable to meet target numbers for all the outputs set for the two year project. Evaluation respondents agreed that the project had clear and relevant objectives to the needs of children with hearing impairment, their families, schools and communities. More children have been enrolled into schools and are regularly attending classes across the four districts where the project was implemented. The strategy to train mainstream teachers was a major achievement, resulting in improved quality of education and hence better performance of learners with hearing impairment. In addition, parents and other community members who participated in the project enabled to better respond to the needs of deaf and hearing impaired children. Support from the Ministry of Education, particularly the Special Education Needs Unit, was evident throughout this evaluation. Key informants contacted acknowledged that the approaches used by the project can and should be replicated by other schools. In hindsight, the project could have been implemented over a longer period of time to achieve the targets, lead to stronger results and increased sustainability.

The difference that the project has made

Within the limited space of time in which the project was implemented (April 2013 to March 2015), significant changes were observed and noted in this evaluation. Most important, the project reached 11,981 beneficiaries, of which 55% (6,637) were direct beneficiaries and the remaining 45% (5,344) were indirect beneficiaries.  Direct beneficiaries included mainstream primary teachers (2,237), parents or guardians of HIC/deaf (1,086), community leaders (1,416), student teachers (202) and HIC/deaf (1,676). On the other hand, of the 5,344 indirect beneficiaries supported by this project, 81% (4,344) were family members of deaf/HICs and the rest (19%, n=1,000) were community members and other teachers. The main changes resulting from this project were reported at different levels. First, learners with hearing impairment were enabled to access education as they were enrolled and supported retention in mainstream schools. Out of 1,676 HIC/deaf children identified and assessed, 75% (1,253) were enrolled in school. A survey of twelve schools indicated that each school had about 13 learners with hearing impairment, i.e. two percent of the total number of children in target schools. A key factor contributing to the first change was improved awareness by parents and guardians on the right to education. Training 1,086 parents/guardians on deaf awareness has contributed to improved interaction between parents, children and school staff. The third change has been improved quality of education offered to deaf/HIC within mainstream schools, through improved teacher awareness and training. Up to 2,459 teachers, of which 9% (222) were student teachers, received practical skills training on teaching hearing impaired learners.  Overall, a large majority of the evaluation survey participants (79%) reported that education for children with hearing impairment have improved over the past two years. In addition, the community has been involved and supporting improved uptake of primary education for all children.

Overall, the short term impacts of the project have been phenomenal. Children with hearing impairment have made good progress academically, socially and emotionally as a result of the ACT Project. Parents who participated in the evaluation felt well supported and especially as they reported that their children’s teachers were considering them as key partners. Learners with hearing impairment also expressed satisfaction at the support they received, including the training they received and the education they accessed. On the other hand, trained teaching staff utilised the skills gained through the training and were developing their confidence working with deaf/HICs. Likewise, school heads and project staff reported an increase in the number of parents and community leaders supporting the project. Overall, this stigma and isolation have reduced, even though still existing as one of the key barriers affecting children with hearing impairment.

How did the project make this difference?

The ACT project was well aligned to government efforts to address various disability challenges in Malawi, with a specific focus on quality education for children with impairments. The strategies implemented by this project were deemed appropriate, especially given the integrated approach used that focused on the various categories targeted by the project.  Besides training teachers and community leaders, awareness raising activities for children with hearing impairment and their parents were noted as key success factors. Based on findings from the endline survey, the most relevant strategies for this project were teacher training on inclusive education (51%), followed by training for hearing impaired children and their families (27%), training for community members (14%) and also support to parent groups (8%). The small income generating activities that were initiated by the project were appreciated, though less sustainable.

The evaluation team also noted areas of improvement with regards to the project strategies. Emphasis was placed on the importance of extending training activities to: several other teachers who were not included in this project; children without hearing impairment who are peers to deaf/HIC; as well as, including secondary schools as part of the project. The need to also address the specific challenged affecting children who are profoundly deaf were highlighted, especially recognizing that most teachers were unable to use sign language or to provide specialist services to children with severe hearing loss. Above all, the evaluation has stressed the need for more robust monitoring, documentation and evaluation processes for future interventions.

Project strengths and challenges

From the evaluators’ perspective, the implementation period was rather short for one to confidently ascertain sustainability of the project benefits, though there is potential. Lessons were drawn from both the enabling factors and the challenges faced during the implementation of this project.

The key strengths of this project have been summarised as follows:

  1. The interventions by CCAP in collaboration with Signal were clearly targeting the needs of an underserved population of deaf and hearing impaired children across the four districts of the Northern Province of Malawi, where the project ACT was implemented.
  2. CCAP, as an organisation, is committed to supporting children with disabilities, which has been a key area of concern to the Presbyterian Church since the mid- 20th Century when Scottish missionaries established a special school for the hard-of-hearing in Chilanga, Kasungu District in Central Region. Likewise, Signal has a clear mission to help deaf people and those with Hearing Loss in the UK and abroad.
  3. The project was built upon effective partnerships with the Ministry of Education in Malawi, CCAP and Signal that has extended over several years.
  4. The financial support received from Comic Relief made the outcomes possible.
  5. This project was built on lessons learned from previous initiatives. Most importantly, the project took an integrated approach focused on empowering local people to fully engage with deaf/HIC and ensure they access quality education.
  6. The ACT project was valued by all stakeholders, especially parents, communities, schools and the Special Education Department as a significant path for the extension of primary education opportunities to deaf and HIC through awareness raising, training and support to the children, schools and communities that were involved in this project.

On the other hand, some weaknesses and challenges were recorded and may need attention in future programming. Amongst the top challenges, the evaluation highlighted the following:

  1. Mainstream teachers who were trained to provide support to deaf/HIC were overwhelmed with work and large numbers of students in their classes. As a result, respondents reported that deaf/HIC were not given the level of support which they required.
  2. Particular learners with severe hearing loss or who were profoundly deaf were not receiving full support from the teachers who received training from the project, given the inadequate infrastructure and environment for effective learning.
  3. The project was mainly focused on primary schools only. This meant that a large number of learners transitioning to secondary schools lacked the kind of support they required post-primary school. It must however be noted that both Signal and CCAP have since identified this gap and fund-raising activities for work in secondary schools began in mid-2014. We recommend that this should be built into future projects.
  4. The project was planned within a relatively short timescale, i.e. two years, which was unrealistic for full implementation of projected targets, results and sustainability prospects.
  5. While the project was flexible, it was rather focused on children with hearing impairment and neglected other types of disabilities which were common in the target schools. On a positive noted, the evaluators acknowledge the subsequent attempts by Signal and CCAP to include other children with special needs in the ACT project activities.
  6. Evaluation respondents stated that there is a great need for assistive devices, such as hearing aids, for children with hearing impairment. Signal was clear that provision of hearing aids was not within their intentions, as they were conscious of sustainability issues, such as the negative effects on children if parents fail to maintain their assistive devices in the long run.  At the same time CCAP was aware of Signal’s position on this and are working independently with the Starkey Foundation to supply hearing aids.
  7. The project monitoring and evaluation system was generally weak. Among other things, CCAP had limited capacity, skills and tools for effectively monitoring the achievements of the project. Closely associated to a lack of a well-designed monitoring and evaluation framework, no baseline was conducted for the project and documentation processes were generally weak.
  8. There was less evidence of the impact of this project in Likoma Island, mainly due to distance and accessibility issues. It would have been ideal if a field office had established in that district.
Conclusions and recommendations

The project has come to an end and key lessons have been drawn. These lessons are critical as recommendations to various categories of people, including parents, school staff, community members, education authorities as well as organisations such as CCAP and Signal.

In brief, the evaluation team recommends:

  1. Parents and guardians, and community: We encourage more parental/community acceptance of deaf/HICs, as well as, support for them to enroll and remain in school. If stigma related to hearing impairment is reduced, there is greater likelihood that other parents would also be encouraged to take their children ‘out of hiding’ and they will support them to participate and achieve in school. We also recommend community leaders to continue raising awareness about deafness and to support local initiatives aimed at increasing children’s access to education and other activities, such as income-generating activities.
  2. Mainstream schools: Teachers and head teachers in supported by the project should continue to provide ongoing support to deaf/HICs in mainstream schools. More teachers from these school would benefit from continuous professional development that is embedded in all future initiatives. Schools may also find it beneficial to identify and engage local sign language and deaf adults from their local communities to support teachers and other children with sign language skills. In addition, training needs to be extended to other children, i.e. those without hearing impairment. Most important, proper documentation of individual targets and progress updates for each child should be made at all times.
  3. Ministry of Education/Government of Malawi: The government should consider necessary improvements in terms of increasing the number of skilled SNE teaching staff in both primary and secondary schools. Where possible, more resource centers for children with special needs should be constructed and fully equipped in all the districts. Likewise, existing resource centres should be supported to increase their reach and effectiveness. We also emphasise the importance of continued investments into evidence based programming for all initiatives, to ensure improved quality of interventions such as the ACT project.
  4. CCAP in Malawi: The ACT Project has brought together a significant amount of lessons on inclusive education for deaf/HICs, which should be shared with others. We recommend CCAP to continue supporting and reaching underserved communities, such as children with hearing impairment. However, we also encourage that projects should extend beyond single disability groups to be more inclusive of all kinds of impairment. In designing future programmes, project strategies should also take into consideration the functional differences between hearing impaired learners, i.e. children with mild to moderate hearing loss and those who are profoundly deaf. While some of the hearing impaired children have access to hearing aids, speech expression and reading; the educational requirements for children who are profoundly deaf are, however, far more complex. We also recommend CCAP to take a bold step in advocacy and influencing key policies around access to education for children with disabilities. A specific issue they may need to focus on is around increasing the allocation of resources for children with special needs, both in and out of school.
  5. We have also highlighted the need to improve on project monitoring, evaluation and learning if the lessons that are being generated by similar projects will have to be shared with other institutions.
  6. Signal: We commend Signal’s support to CCAP and encourage them to continue playing a facilitation role that links them with other potential funders, while providing technical support to organisations such as CCAP. We also commend Signal Trustees for the recent approval to keep their core element of hearing loss and deafness but also expand to other special needs. We particularly recommend Signal to continue mobilising resources for the consolidation of lessons learnt and scale up of the ACT project.
  7. Funding partners, such as Comic Relief: Comic Relief and other funders are encouraged to fund follow up project initiatives and support the work of Signal and CCAP. In particular, we have highlighted the need to expand the project to all children with various forms of impairment and to also scale out the activities to secondary school children. Continuous development programmes for teachers would also be a key priority in order to sustain the efforts of this project.