Volume 134, June 2014, Pages 17–28
Development of a participatory tool for the evaluation of Village Animal Health Workers in Cambodia
- Under a Creative Commons license
Open Access
Highlights
- •
- We developed a criteria grid for the evaluation of Village Animal Health Workers.
- •
- Participatory approach allowed actors to develop their own indicator of success.
- •
- Our tool allow actors to evaluate their own situation, skills and capacities.
- •
- Our tool produces useful quantitative results for decision-makers.
- •
- Participatory tools enhance the probability to take actions after evaluation.
Abstract
In
countries with a lack of primary care systems, health workers are of
crucial importance to improving the delivery of health and animal health
services at community level. But somehow they are rarely evaluated and
usually with a top–down approach. This is the case in Cambodia, where
thousands of Village Animal Health Workers (VAHWs) have been trained by
the government, and where no standardized evaluation tool is available
to accurately assess the situation.
Based on methodology developed by the French NGO Agronomes et Vétérinaires Sans Frontières
(AVSF) in Madagascar for farmers’ association evaluation, we developed
our own participatory methods to collect information about the VAHW
context and build a criteria grid for their evaluation. In this
framework, several participatory approaches were used such as problem
trees, semi-structured interviews, pair-wise ranking and focus groups.
The grid was built with the help of relevant stakeholders involved in
the animal health system in Cambodia in order to (i) identify VAHW
functions; (ii) set up criteria and associated questionnaires, and (iii)
score the grid with all the stakeholders. The tool was divided into
five categories of evaluation criteria: sustainability, treatment,
production, vaccination and disease reporting.
Our
approach looked at local indicators of success developed and used by
VAHWs themselves, which should lead to better acceptability of
evaluation. This method gave priority to dialog aiming to engage
decision makers and other stakeholders in a mutual learning process and
could be applied in other countries to develop trust between health
workers and official service representatives as well as to foster
corrective action after evaluation.
Graphical abstract
Keywords
- Participatory tool development;
- Criteria method;
- Community-based evaluation;
- Health workers;
- Cambodia
Abbreviations
- AVSF, Agronomes et Vétérinaires Sans Frontières;
- CV, communal veterinarians;
- DAHP, Department of Animal Health and Production;
- DV, district veterinarians;
- FAO, Food and Agriculture Organization;
- FO, Farmer Organizations;
- GDP, gross domestic products;
- MAFF, Ministry of Agriculture Forestry and Fisheries;
- NaVRI, National Veterinary Research Institute;
- NGO, Non-Governmental Organization;
- PAHP, Provincial Animal Health and Production;
- PE, Participatory Evaluation;
- VAHW, Village Animal Health Worker;
- VRC, Vétérinaires Ruraux du Cambodge
1. Introduction
Agriculture
represents the most important economic sector for Cambodia, with the
livestock sector accounting for around six per cent of gross domestic
product (GDP) (Harding et al., 2007). Most of the livestock is produced in the smallholder farming system (Harding et al., 2007).
However this sector is characterized as a low income-generating
activity with a high level of morbidity and mortality of animals due to
the farmers’ poor feeding resources, lack of efficient local veterinary
services, limited technical skills and access to medicine (Chheng, 2009a).
To overcome these challenges, in the early 1990s the Cambodian
government started training volunteer farmers to provide animal health
services at village level: Village Animal Health Workers (VAHWs) (Burgos et al., 2008).
The farmers are selected within their own community and trained in the
basic techniques of veterinary medicine. They are in charge of providing
advice, technical assistance and animal healthcare services, including
vaccination, treatment and husbandry practices (Burgos et al., 2008 and Chheng, 2009b).
The first wave of training was largely implemented with the help of
external support (NGOs and FAO). Development objectives were targeted
through solid theoretical training and practical exercises (Benzerrak et al., in press) in order to improve livestock production (CelAgrid, 2007) but no formal evaluation was carried out afterward.
After
the H5N1 Highly Pathogenic Avian Influenza crisis in Cambodia in 2004, a
second wave of training was introduced. These sessions targeted public
health security and were undoubtedly implemented in response to
international pressure. The government objective was to have at least
one trained VAHW per village to improve poultry disease surveillance and
ensure early detection (vice-chief of the Department of Animal Health
and Production, DAHP, personal communication, 2011). To reach this
objective, the government delegated training to diverse NGOs such as Agronomes et Vétérinaires Sans Frontières (AVSF), Heifer International or Care International, within specific projects funded by the FAO ( CelAgrid, 2007).
This strategy has led to a massive training period, with marked
heterogeneity in regard to training contents, participant selection
processes and training session duration. Up to now most VAHWs have
received training (or refresher training sessions) focusing on AI ( Wilsmore et al., 2010). This strategy has diverted attention away from other diseases ( Wilsmore et al., 2010).
In Cambodia most VAHWs are men, with an average educational level grade of 7.8 (CelAgrid, 2007).
Their only source of income as VAHWs comes from the farmers who pay for
their services, although most VAHWs practice other activities at the
same time (e.g. rice production) (CelAgrid, 2007).
They work in close collaboration with village chiefs to implement
animal health campaigns and activities recommended by the Ministry of
Agriculture Forestry and Fisheries (MAFF), and NGOs (Burgos et al., 2008).
They are not government agents, but are regarded as representatives of
the DAHP at village level and it is mandatory for them to report
suspected outbreaks of notifiable diseases, even if they receive no
compensation for their involvement (Wilsmore et al., 2010). This task is made difficult by the fact that there is no compensation scheme for farmers after major outbreaks (Wilsmore et al., 2010).
So far 12,000 VAHWs have been trained (http://www.fcfd-cambodia.org/avsf/avsf_successes_en.html)
but according to the French NGO AVSF several issues within the system
remain: the heterogeneity of their skills, due to (i) the high number of
trainers, (ii) the diversity of organizations providing training, (iii)
the inconsistent selection criteria for VAHWs and (iv) training
contents (AVSF, personal communication, 2011) and the fact that each
year more and more VAHWs cease their activities. Indeed, 7.5% ceased
activity in 2010 (vice-chief of the DAHP, personal communication, 2011)
and probably even more. However, the term ‘activity’ remains unclear and
there is no information about the kind of activity the VAHWs have
ceased: livestock and poultry production (first training wave), and/or
animal disease reporting (second training wave). The status and
definition of VAHWs also remains unclear.
The
main objective of VAHWs is to help the veterinary services (SV) to
achieve the priorities of national government by improving animal health
and welfare, as well as global human health (OIE, 2013).
These services should be regularly evaluated in order to gage their
current level of performance and identify any gaps and weaknesses in the
system to make recommendations for improvement. Evaluation basically
refers to the collection, analysis, interpretation and reporting of
information, in order to judge the success or failure of an intervention
or a project (Rice and Franceschini, 2007).
Most evaluation processes are conducted by external stakeholders, yet
the project participants, who are directly concerned by the evaluation
outputs, are not taken into consideration either in the process of
question formulation or in data collection (Rubin, 1995).
Bradley et al. (2002) recommends using participatory evaluation (PE) as
this facilitates mutual learning, it helps participants evaluate their
own needs and analyze their own priorities and objectives thus leading
to better acceptability. PE can be defined as applied social research
that implies interactions between stakeholders (Garaway, 1995 and Lahai, 2009), focusing on the understanding of local realities and on continuous learning (Rice and Franceschini, 2007).
This method leads to stakeholder empowerment in the process, which
could improve the sustainability of health interventions. The use of
such developmental evaluation (DE) has been recognized as a way of
supporting adaptive learning, leading to a deeper understanding of the
stakeholder's problem/opportunity, resources, and the broader context (Dozois et al., 2010).
Moreover, a key principle of DE is to ‘help stakeholders surface and
test their assumptions, articulate and refine their models, extend their
understanding and cultivate a culture that supports learning’ (Dozois et al., 2010).
One
example of the development of a PE tool was initiated by AVSF in
Madagascar for the assessment of Farmer Organizations (FO) and then
adapted in Cambodia in the framework of the Komrong Daikou project (Gennet and Martin, 2012).
This tool is implemented in four stages: (i) define evaluation criteria
through workshop(s), (ii) finalize criteria in guidelines and
communicate these to farmers, (iii) evaluate and score criteria in the
field and (iv) link scoring results with suitable and available
technical and financial supports (Gennet and Martin, 2012).
Through
this experience, AVSF highlighted the fact that this was a very
simulating process involving farmers’ perceptions, action and
commitment. Moreover, results have shown the great potential of the
tool, with an increase of FO activities, self-initiatives and management
skills (Gennet and Martin, 2012).
According to the identified needs, and following the success of this
project, AVSF, with the support of researchers from CIRAD, decided to
develop its own tool using the same participatory methodology: the
involvement of key stakeholders in defining evaluation criteria and
scoring these criteria.
The
main objective of our study was the participatory development of a tool
to assess VAHW performance and level of activity. Our aim was to
initiate a process of participatory evaluation and to harmonize
perception of the needs, expectations and responsibilities of VAHW among
the different actors and to ensure that the results are used for
change. To build this evaluation tool, a series of meetings was arranged
targeting VAHWs as well as the main stakeholders working in animal
health systems in Cambodia. The work was carried out in several steps
leading to a criteria grid to evaluate VAHW sustainability and
capacities (Table A1).
2. Material and methods
2.1. Study area
Two
provinces were selected according to the presence/absence of diverse
NGOs working in the field of animal health and in collaboration with
VAHWs. Indeed we assumed that their involvement in a community project
in the province would impact not only their answers, due to the
perception of their activity by NGOs, but also their commitment in the
development process of the criteria grid. Moreover, the presence of an
NGO in the area undoubtedly impacts VAHW sustainability. The objective
of this methodology was to detect the different perceptions of VAHWs
according to the context in which they operate. Prey Veng province was
thus chosen because of the historic presence of AVSF in the area, while
Svay Rieng province was chosen because no NGO was working with VAHWs to
our knowledge. Because of time constraints only one district was
selected per province. In Prey Veng province, Ba Phnom district was
chosen due to the presence of an active, locally recognized association
of VAHWs supported by AVSF. In Svay Rieng province, the district of
Kampong Rou was selected because VAHWs were more isolated and received
no support from NGOs.
2.2. Targeted population
To
improve the relevance of our tool it was fundamental to triangulate the
various levels and sources of information. Therefore we decided to
integrate the points of view of a maximum number of stakeholders
involved in the animal health services or with specific knowledge or
expectations related to VAHW activities and to select local and national
actors. In each district three groups of VAHWs were selected by the
District Veterinarian: active VAHWs (officially trained and still
exercising their activity at least in their own village), inactive VAHWs
(officially trained but no longer exercising their activity or never
did) and Commune Veterinarians (CVs). CVs are the unofficial
representatives of VAHWs at commune level, appointed by the DV or by the
VAHWs of the commune; their role is to facilitate communication with
government representatives. The DV is a DAHP representative at district
level; he is in charge of implementing and enforcing animal health
regulations. Local authorities, policy makers, ministry representatives,
local and international institutions (Food and Agriculture Organization
of the United Nations and NGOs) were also involved. They were invited
to several meetings in the framework of the study and were involved in
developing the tool by demonstrating their expectations and perceptions
of VAHW activities. Private sector representatives (from Medivet, Bunlay
Kry Progress and CP) working in relation with animal breeding, animal
production and animal health were contacted but were not available at
the time of our study. Per diem was provided to all stakeholders who
attended the meetings.
2.3. Design of the criteria grid
The design of the criteria grid was conducted in six phases as described in Fig. 1,
from March to November 2011. Meetings held in the field were conducted
in Khmer and directly translated into English, while the workshops in
Phnom Penh were conducted in English and translated into Khmer. The
meetings held in the field were coordinated by two researchers, one from
the Royal University of Agriculture of Phnom Penh and one from CIRAD.
These facilitators acted as catalysts and tried not to take part in the
discussions.
- Fig. 1.Description of the phases implemented for the design of the criteria grid. (1) Use of the identified functions as a basis to develop evaluation criteria. (2) Use of the developed evaluation criteria to elaborate an evaluation grid. (3) Implementation of the elaborated criteria grid in the field. (4) Modification of the criteria grid according to field implementation. a VAHWs: Village Animal Health Workers. b CV: Commune Veterinarians. c DVs: District Veterinarians. d CC: Commune Council. * Representatives from the Food and Agriculture Organization, from the National Veterinary Research Institute, the Ministry of Agriculture, Forestry and Fisheries, and from Non-Governmental Organizations.
2.3.1. Phase 1: Identification of VAHW functions
During
the first phase, VAHWs and CVs from the same district were brought
together for a half-day meeting. The objective was to help them identify
and formalize their functions in their own community. A function was
defined as an activity carried out by a VAHW at village level but also
any action related to requirements from local or national authorities.
The identified functions would then serve as a basis to develop the
evaluation criteria. This method initiates thinking about the VAHWs’ own
perception of their roles and duties. Three half-day meetings were
conducted in each district targeting three distinct groups of
participants (cf. Section 2.2). We invited 10 participants to each meeting.
To
foster more effective group discussion and collect as much information
as possible the Metaplan method was chosen and adapted to the context.
This method saves time and ensures the involvement of all participants
in the group (©Metaplan GmbH, 2003).
Each participant was provided with sheets of paper and wrote down one
of their functions per sheet. Once collected by the research team, the
papers were read out to the attendees and functions related to the same
topic were grouped into categories selected by the participants. Having
identified the categories, the research team encouraged discussion about
all VAHW functions. The facilitator was careful not to take part in the
debate but merely to guide it in order to collect raw information. At
the same time, a problem tree was used to define the VAHWs’ work context
and main constraints. The trunk represents the main problem faced by
the participants, the roots, the causes or factors and the branches, the
effects or consequences. This method is considered as heuristic as it
helps to identify, prioritize and visualize problems (Veselý, 2008).
The facilitator asked attendees about the main difficulties encountered
by VAHWs in their activities. After a group discussion the participants
agreed on one main difficulty, which was identified as the main
problem. The facilitator had further discussions with the group to
identify the main causes and consequences of this problem.
At
the end of each meeting the research team asked for two volunteers to
be involved in the next steps of the project. A total of 12 volunteers
were invited: four active VAHWs, four inactive VAHWs and four CVs.
2.3.2. Phase 2: Identification of evaluation criteria by VAHWs
During
the second phase, a half-day meeting was held, bringing together the
volunteers from previous meetings and the DVs of the targeted areas. The
objective was to define criteria to evaluate the functions identified
during the first phase.
The
research team chaired a discussion in which participants were invited
to discuss and validate the set of categories and functions previously
identified. Participants were then divided into four heterogeneous
groups each including active, inactive, CV and DV. Each group worked on
one specific category only. For each function, they had to (i) identify
the ideal ways of performing it, (ii) identify the relevant stakeholders
to interview to check whether this related function is well performed
by the VAHW (e.g. villager, village chief), and (iii) work out the
questions to be asked to assess VAHWs’ performance level. Once the list
had been drawn up within each group, one member of the group presented
their work to the other participants. To validate, add or remove
evaluation criteria open discussions were led by the facilitator.
2.3.3. Phase 3: Complementary meeting to identify VAHW functions and evaluation criteria
In
the third phase, a half-day meeting was held in Phnom Penh gathering
together the stakeholders directly involved in the training and
coordination of VAHWs (Fig. 1):
representatives from the FAO, from the National Veterinary Research
Institute (NaVRI), from the MAFF, and from NGOs. As in the previous
phases, the objectives were (i) to define VAHW functions, (ii) define
evaluation criteria and (iii) draw up the associated questionnaire. To
save time, the research team used the categories identified during the
first phase as a basis for the meeting. The same approaches and
participatory tools were then used (Fig. 1, phases 1 and 2).
2.3.4. Phase 4: Validation and scoring of the evaluation criteria
All
the participants who joined the previous meetings were invited to a
one-day workshop in Phnom Penh which was conducted in three stages. (1)
During the first part of the meeting, the facilitator described the
evaluation criteria in the provisional grid one by one with the
associated questionnaires and asked the participants for their
validation. Requested modifications were made following the agreement of
all the attendees directly on the spot. (2) In the second phase,
categories were ranked according to their relative importance in VAHW
evaluation using a pair-wise ranking method. This is a slightly more
complex ranking system whereby each item is compared individually with
all the other items one-by-one in a consensus-oriented manner (Ameri et al., 2009).
This approach is considered more reliable than simple ranking as it
imposes the consideration of every possible relationship (AFENET, 2011).
The exercise was carried out in two steps: first by asking VAHWs and
CVs for their point of view and then by asking other attendees for
validation. The number of times a category appears on the comparison
table can be correlated to its rank and thus to its weight in the
evaluation. Participants then allocated the 100 points of the grid to
these categories according to their relative importance in VAHW
evaluation. (3) In the final phase four heterogeneous focus groups were
formed, each helped by a facilitator, to work on weighting the
evaluation criteria. Indeed, each category is composed of several
evaluation criteria that had to be weighted according to their relative
importance in VAHW evaluation. Points previously allocated to the
categories were attributed to the evaluation criteria by discussion
within the group. Each group worked on each category. The results were
then collected, means of the scores calculated and presented to the
participants in order to lead to an agreement. At the end of the day the
first complete list of scored evaluation criteria was accepted by
common consensus, with a list of stakeholders to interview during the
field evaluation and its associated questionnaire.
