 
Week 3 - Outcomes and Standards
|  | Program Evaluation in Schools |
| |
Listing of Papers | |
MS NISHA HEATH
Victoria, Australia
IN A CLIMATE of ever-increasingly-rapid change and economic rationalism, schools are often called upon to measure their efficiency and effectiveness. Program evaluation is a useful way to do this. In this article, I will provide a detailed explanation of the main elements of program evaluation, including evaluation planning, the difference between qualitative and quantitative data and information dissemination. There are many forms of evaluation, but interactive evaluation is the form most commonly used by schools and my article will focus on that form.
Schools use different program evaluation approaches for various purposes. Therefore, some different approaches to interactive evaluation will be summarised, including action research, developmental evaluation and empowerment evaluation. I will outline four evaluations that have occurred in schools recently using each of these three approaches to evaluation.
Program Evaluation
Evaluations provide people with a means of examining issues in objective ways. Evaluations fully define the object under scrutiny, the methods by which information about the object will be collected and ensure that the most relevant stakeholders are the people who receive findings.
Evaluations are about planning carefully, reflecting critically and justifying reasons for selected courses of action (Owen, 1999:1). Program evaluations can occur before, during or after the program has been initiated. Different kinds of evaluation can help at all stages of program implementation, and forward-thinking organisations could use evaluation continuously to help with change management and continuous improvement.
Program evaluations consist of four stages:
- planning;
- collecting and analyzing evidence about the object of the evaluation (evaluand)
- disseminating findings based on this evidence to audiences; and,
- understanding or making judgments and decisions (Owen, 1999:20).
People within, or external to organisations, can be evaluators.
Stakeholders are the people who have an interest in the evaluand. One of the initial tasks for evaluators is to determine which stakeholders will be the users of the findings of the evaluation. People are much more likely to take on changes as a result of decisions made due to evaluation findings if they are involved in the evaluation. Participants in evaluations will feel some ownership of the findings due to the work they have put in and their enhanced understanding of the evaluation process as a result of that work. Therefore, key stakeholders are the people who should have direct involvement in the evaluation process.
1. Interactive Evaluation
Interactive evaluation is a form of evaluation often used in schools because it directly involves the key stakeholders in the evaluation process. Three approaches within interactive evaluation that I believe would be particularly useful for schools are action research, developmental evaluation and empowerment evaluation.
Action research involves developing a plan of action, implementing the plan, observing the effects of action, and reflecting on these effects as a basis for further planning, and subsequent action (Kemmis, 1985). Action research is compatible with processes that already occur within schools, and has a cyclic implication that would facilitate schools engaging in continuous improvement.
Schools adopting action research would need to ensure that staff who were not directly involved in the evaluation process were nevertheless briefed about why it was occurring, and that data collection occurred in such a way that disruption to student learning was minimal. There are sometimes problems with ensuring that the data collected is reliable when using an action research approach, so the evaluation team would need to ensure the use of necessary rigor.
During developmental evaluations, the evaluator is part of a team whose members collaborate to conceptualise, design and test new approaches in a long-term ongoing process of continuous improvement, adaptation and intentional change (Owen, 1999:228). Developmental evaluation sits well with the school environment, where the organisation is continually evolving to meet the needs of the changing staff and students as they move through the school. Choosing an evaluator familiar with the school culture and philosophy is important in this situation, to ensure that the evaluation team is able to work effectively.
Empowerment evaluation involves program staff, participants and evaluators coming to a consensus about their mission, vision and expected results; using current processes as a baseline to plan for the future; and developing strategies linked to the attainment of goals. Empowerment evaluation has self-determination (the ability to chart one's own course in life) as its basic tenet (Owen, 1999). Surely, one of the main purposes of schools is to inculcate students with self-determination in order that they leave their school ready to deal with the thrust and parry of the real world. Thus, becoming an organisation that can adopt this tenet is an important consideration for schools themselves.
2. Planning a Program Evaluation
Negotiating an evaluation plan with key stakeholders achieves several tasks. The plan:
- sets direction;
- determines key issues which need to be addressed;
- determines the audience who will receive and use the findings; and,
- determines, in light of the target audience, the best way to present the information in the findings.
'Up-front' planning and negotiation, and involvement throughout the evaluation of key stakeholders, build in support for the evaluation effort. As there are often a diversity of views amongst stakeholders about the evaluation's purpose, it is important to determine which stakeholder groups' agendas to address early in the investigation, and negotiate the evaluation design with these one or two groups. The main elements of the evaluation design to agree upon are key issues, the timeline and the budget (Owen, 1999). It is also important to discuss the nature of the stakeholders' involvement in each stage of the evaluation, so that they are aware of what tasks they will be required to complete, and when.
3. Qualitative and Quantitative Data Collection and Analysis
Data collection is the collection of information by evaluators to produce findings. Broadly, qualitative data collection refers to the practice of describing information, using language; and quantitative data collection is that which uses more scientific methods and numerical indicators. The former originates in humanistic research and the latter in scientific research traditions. There are advantages and disadvantages of using either method.
When statistics and numbers are the main information sources, data is usually reliable and relatively easily analysed. However, context-specific and wider organisational issues are often ignored. When information is collected using more descriptive accounts that can address 'grey areas', selecting procedures exhaustive enough to provide a useful analysis of data collected can be a problem. Additionally, analysing qualitative data is often time-consuming. A combination of both data collection methods will ensure scientific rigor and humanistic contextual considerations.
Whatever methods are used, multiple sources of data should be used as a check on the trustworthiness of the data and consequent hypotheses. Various taxonomies of data collection methods contained in research textbooks help in linking appropriate data collection techniques to specific evaluation questions (Hopkins, 1989).
4. Disseminating Findings, Decision-Making
The last stage of an evaluation involves disseminating findings based on the evidence relating to the evaluand to audiences, for use in understanding or making judgments and decisions. Evaluators could meet with the stakeholders and the commissioners of the evaluation and use a Powerpoint demonstration to present their findings. Communicating an issue in person is an effective way to ensure that people actually receive the information. This method also provides an opportunity for evaluators to answer questions that people may have about the evaluation. The findings could take the form of a document with tables of qualitative and quantitative data included. In this case, it would be important to ensure that all of the relevant people receive the document, and that it is actually read by those people. People with a stake in programs are not always the people who commission evaluations. Therefore, it is important to present findings in such a way that they are useful and understandable to all concerned.
Interactive Evaluation in Schools
Hopkins argues that evaluation enables teachers to develop their professional abilities as educators, to become collectively more self-conscious and systematic about the exercise of their own and their colleagues' professional judgments, and to establish an infrastructure at the school level for so doing (1989:195).
The planning stage of evaluations is of utmost importance, and such proactive behavior is not always followed in schools. Many programs and policies are introduced without planning and without involving key stakeholders in schools. Thus, teachers and administrators would learn a lot from just the initial stages of preparing a participatory evaluation. Some schools do already use interactive evaluation for various purposes and four examples are listed below, in which schools use the approaches of developmental evaluation, action research, and empowerment evaluation.
Country Education Project (CEP)
The CEP received a grant to fund professional development programs for teachers in rural centres in two States. Evaluators conducted observations and interviews at sites, that formed the basis of a paper, which was then discussed at meetings at which changes to the program for subsequent implementations were decided (Owen, 1999:230).
Developmental evaluation was used at these schools to facilitate professional development for teachers. Although the evaluators collected and analysed the data, they interacted with a team from the schools and, based on this interaction, decisions were made about the future direction of the project.
Girls and Mathematics Science Project (GAMAST)
GAMAST was a nationally-commissioned project designed to increase participation of girls in mathematics and science subjects in schools and other educational institutions. The evaluators worked in a cyclical mode of issue-setting and evaluation. At the beginning of each cycle, a meeting of staff and evaluators decided on an issue for attention. Evaluative feedback was brought to each following meeting for discussion to enable project staff to plan the next round of action (Owen, 1999).
Action research was undertaken at this school in order to make some decisions about how to raise the participation rates of girls in mathematics and science subjects. A plan of action was developed, and after implementing the plan and reflecting upon the effects of the action, new issues were raised and the cycle continued.
Saturn School
An internal evaluation occurred at Saturn School, in Minnesota in the United States. The school adopted an innovative program in which the roles of teachers, pupils and the parent community were redefined. The curriculum of the school was developed from the information gathered from the teachers and students. The curriculum was changed every ten weeks. An evaluator was employed to document the implementation of the curriculum and to provide systematic feedback to assist in its development (Owen, 1999).
At the Saturn school, students were included as important stakeholders during an empowerment evaluation process that helped to create a whole-school-community approach to curriculum. Staff, students and evaluators came to a consensus about their mission, vision and expected results and were able to generate and document a curriculum that could develop and grow with the school.
Elementary School Mathematics
A 'critical friend' (evaluator) took part in an evaluation of an elementary school mathematics program. The evaluator provided objectivity and rigor, whilst thinking along similar lines in sympathy with the philosophy and direction of the school (Owen, 1999:235). This elementary school carefully chose their evaluator who was working as part of a team when conducting a developmental evaluation in order to improve their mathematics program.
Programs change continually to meet the needs of students, to update the content of the curriculum, to ensure excellence of resources and as a result of staff turnover. Curriculum, course, teacher, project and program evaluations are some of the ways in which schools can use evaluation to encourage continuous improvement in conjunction with such changes. The use of internal, interactive evaluations brings evaluation into the decision-making communities of schools (Norris, 1993:101,145).
Conclusion
Program evaluation can assist schools in objectively examining their practices in all areas. There are many forms of, and approaches to, evaluation. In this article, I have argued that interactive evaluation is the most appropriate form to use in schools, and that the approaches of action research, developmental evaluation, and empowerment evaluation are particularly useful. Schools are already using evaluation to assist them in a number of areas, and four examples of such use have been discussed in this article. An ongoing cycle of evaluations could enable schools to plan an exciting future using their best past practices.
_____________________________________________________________
ABOUT THE AUTHOR
Ms Nisha Heath is currently Head of Junior School Music at Penleigh and Essendon Grammar School, overseeing the musical development of 900 students, with the help of a team of 34 music staff and the Music Director. She started the position this year. Her career spans nine years and she has taught music at various primary schools in both the independent and state school systems.
Several years after the completion of a Bachelor of Music Education from the University of Melbourne, she developed an interest in leadership and returned to the University of Melbourne part-time to study for a Postgraduate Diploma of Educational Administration. Upon completion of the course, she chose to continue studying and she is currently completing a Masters of Education.
She is a regular writer for 'Directions in Education', and had an article published in 'The Practising Administrator', volume 21 no.2, in May 1999, 'Women and Leadership in the 1990s'.
_____________________________________________________________
REFERENCES
Hopkins, David. (1989). Evaluation for School Development. Great Britain:Open University Press.
Kemmis, S. (1985). The Action Research Planner. Geelong: Deakin University Press.
Norris, Nigel. (1993). Understanding Educational Evaluation. London: Kogan Page Ltd.
Owen, John M. (1999). Program Evaluation: Forms and Approaches. Australia: Allen & Unwin. |
|
| | |
