Chapter 7: HOW TO DO EVALUATION

Rob McBride and John Schostak

We have a great deal to learn about the 'insider evaluator', that is, the teacher who is evaluating aspects of her own school. Nevertheless, this chapter is written with the insider evaluator in mind rather than the practising evaluator who works on a commission basis. A major difficulty facing the insider is her familiarity. Of course in one sense this is an advantage in that she knows who does what, where they can be found, something about their history and so on. On the other hand she carries with her a set of understandings and expectations, deriving from what she is used to, that the outsider might not. To some extent it is necessary to adopt an outsider view in order that creative and unusual questions can be asked and suggestions made.

As you go through this chapter it should soon be clear that the qualitative evaluator is in a sense an intruder who seeks information that is not always public and is often personal despite being professional knowledge. The knowledge that the evaluator accumulates gives her a measure of power over her correspondents which has to be used very carefully as the debate she seeks to generate emereges. In both collecting data and using it, it is essential that her relationships with the correspondents is a trusting one. From time to time misunderstandings can quickly surface and then become sources of friction unless handled carefully. Accordingly the evaluator has to reassure respondents; she has to concentrate on creating trust; she has to reveal herself as prepared for difficulties, farsighted and competent; she has to be tolerant of the concerns of others about her work; and she has to be determined to produce reports and/or statements which influence the decisions that her colleagues will take.

The evaluator should begin by having a strong grasp of the ethics of evaluation and prepare a a document which outlines her principles of procedure. Let us turn to these first.

A. THE ETHICAL PRINCIPLES OF ENGAGING IN RESEARCH

These are vital. Research in schools, as in other institutions, is a community action. In order to maintain a positive ethos the rights of all the community - staff, children and parents - have to be respected. For this reason it is worth considering the principles under which the research will be conducted.

It is worth repeating that these are general principles relevant which can be modified or form the basis of guidelines for any project:

Researchers have to negotiate their way through their research. They have to negotiate access, i.e. they have to ask colleagues and others if they will agree to be researched and if they will give up time for interviews and similar activities. If access is agreed and an interview given, steps should be taken to ensure that the account is fair, accurate and relevant.. This is called the negotiation of accounts. There are various degrees of negotiating accounts. A strong form occurs when complete manuscripts are returned to interviewees but in most cases this is a daunting and impracticial task. More often, guarantees of confidentiality and anonymity allow the interviewer to use data from an interview. In this case the interviewer has an ethical duty to act responsibly. Alternative strategies include:

- showing complete transcripts only to those who are most easily recogniseable.
- showing edited parts that might be included in a final report.

The negotiation of boundaries rests upon the respect which should be offered to all participants, especially those without power. There maybe a number of different perceptions that have to be pieced together. All of these should be considered. Should a respondent assert that the testimony of X is basically flawed the researcher will have to ask why. Some interviewees may use part of an interview to carp about a colleague and the response could be that the interview is concerned with issues not people. Plainly there is a narrow line between the two.

Research within schools can too easily be abused. Within the community of the classroom, teachers are powerful figures in the lives of children. Similarly, within the community of the school Headteachers are powerful figures in the careers of their staff and their pupils. Such power relations, if used in the process of research will lead to abuse. Knowledge is power. And knowledge of other people is power over people.

An essential question which must be asked by all those who want to engage in research is: Why and for what purposes is the research to be carried out? Basically, this is an ethical question. In answering the question, one must ask whether anyone's interests, needs, feelings will be damaged or hurt. One must ask also, if anyone will profit by the research and in what way.

The central force of the ethical principles outlined here is concerned with the collection of data. It has been pointed out that there is little reference to the action or change that maybe anticipated to emanate from the research. The above question in bold type suggests that researchers should pay attention to this question as does, to some extent, the principle of empowerment. The value that qualitative research attaches to human decision making is an indication that control of action should lie with individuals. Equally important is the conviction or assumption that the engagement of practitioners in researching their own practice will prompt action. This is often treated as empirically or historically demonstrable by established qualitative researchers [see MacDonald in Rudduck, 1991]. As I have pointed out above, qualitative researchers are generally well aware of the importance of practitioner engagement as an essential factor in change.

Nevertheless, we live in an age of information gathering. What will happen to the data? What rights do people have in relation to such data files? Is research just another process of gathering information on people? These are uncomfortable questions, but must be asked by the researcher.

B. PRINCIPLES OF PROCEDURE

It is likely that each study will require a slightly different set of principles. It is not important to show each interviewee the document before you begin, indeed, this can create too much formality at the start of an interview. Writing up such a set of principles of procedure does help the evaluator to put ethical isues at the front of her mind before she begins. It can also be used to show any doubting colleagues that a study is well thought out and constituted on a considered basis.

As a guide here is a copy of a set of principles recently used in a study of museums and schools.

Developing Schools' Use of Museums. Principles of Procedure of the Evaluation.

These are the principles which will guide us in the evaluation.

1. Access - the evaluators reserve the right to approach anyone with an interest but will proceed only with the permission of the relevant teacher, school, curator or museum.

2. Independence - no participant in the project will have power of veto over the content of the report.

3. Disinterest - in the evaluation report the evaluators will attempt to represent the range of viewpoints encountered rather than present their own views. They will subsequently draw up a set of recommendations for future action after dsicussion of the evaluation report with the sponsor.

4. Negotiation of Accounts - where possible, the accounts of participants will be negotiated with them to ensure they are fair, relevant and accurate. Negotiated accounts will be regarded as having the endorsement of those involved.

5. Confidentiality - where possible, all accounts will be treated as confidential. No documentation will be examined without permission and no data will be collected 'off-the-record'. Any attributed quotations will be used only wth the permission of the individual concerned.

6. Circulation - reports will only be circulated after accounts have been endorsed as fair, accurate and relevant. The evaluators accept that the sponsor has the right of first refusal on wider publication.

These principles have been taken from a small commissioned evaluation. Had the project had a greater 'visibility' point 6, for example, would have been more concerned about publication.

Having put down these foundations the evaluator is now in a position to begin.

C. IMMERSION

This step is not as unproblematic as it might seem. The evaluator may believe that in her school it would be better to withdraw children for certain lessons or that a certain child is beyond redemption as far as the school is concerned. The evaluator has to try and clear her mind. Of course this is not completely possible but it has to be attempted. Objectivity is not an option, for this cannot be achieved. It is more a question of saying to oneself, "Well at last I have an opportunity to sit down and have a look at this." It is a sort of end of term feeling when you can finally sit down and read a book you have had your eye on or take a walk you've wanted to take.

At this stage observation is terribly important. The evaluator might consider herself a 'fly on the wall' or 'looking through a crack in the fence'. While this is going on the creative thought can begin. One might ask if one observes a confrontation between pupil and teacher, "Is this necessary? Is there an entirely different way of doing this?"

Gradually, in the course of time, the erstwhile evaluator will begin to feel an issue emerging, hopefully in a form she has not perceived before. During this time it would not be helpful to pretend that all is under control and the evaluator has it all tied up. It is more constructive to admit to being unsure, of finding out. The evaluator will often find that her colleagues will offer information and this is what she needs.

D. DOING INSIDER EVALUATION

Following MacDonald's comments on democratic evaluation, the evaluator will almost certainly find that different individuals and groups will 'see' the same situation in different ways, that is there are multiple realities according to the values people hold. Teachers who want their children seated quietly in rows maybe disconcerted at pupils walking around a room while another teacher who wants children to have more freedom may interpret the same activity as constructive and positive. It is the evaluator's task to articulate each one of these perspectives.

Some of these 'stakeholders', as they are sometimes called, will have to be interviewed in order that the evaluator might begin to make sense of her observations. Has she interpreted the views and actions of her colleagues in a way that they themselves are happy? It is often surprising to find just what one's colleagues believe.

While these activities are taking place the evaluator has to begin to ask some fundamental questions about her evaluation, such as:

- What is it that I want to know?
- Who are the main players in this scenario?
- What are my initial hypotheses?

Data is beginning to pile up. Find time to read notes and listen to tapes, if you have them. This will help you familiarise yourself with your data. It is imperative that data is carefully stored and kept in a way that you have easy access. Filing must be clear and useable.

Soon the first tricky step needs to be taken. As has been mentioned above a major part of your evaluation is the formative role you play. You may go to a departmental meeting, for example, and suggest that some children feel that they are not being sufficiently stretched or, alternatively, being stretched too far. You will need to say this tactfully. One way is to ask a question which takes the conversation into this area. So you might say:

"Do you think that SEN children are always stretched enough?" and later "How is it that you stretch the children that you teach?" Once you have asked these questions you will need to look carefully for a space to keep bringing the conversation back to this subject until the interviewee decides to ask herself the question that you are interested in. Sometimes interviewees will ask you what the children think and you can then tactfully begin to enter a discussion about the subject. It is this activity of acting as 'information broker' that is at the heart of evaluation. You may well find that some people get 'wound up'. Your task is to keep a cool head. Remember that your job as evaluator is to listen to everybody, indeed, in this respect impartiality means that you are willing to listen to everybody. At the outset you will probably find that , especially as an insider, you have views which may favour one group or another. If you are prepared to listen for long enough you will more than likely at least be able to see other sides of arguments. Your own views may become less important to you.

You will almost certainly find some people and some areas disconcerting. It is sometimes helpful if you think of yourself as a servant or at least as someone who is trying to help her colleagues improve their practice. This subservient role can be further enhanced if you act as a source of useful, especially researched, information. To be able to converse with a colleague on their own ground as it were, can increase their confidence and trust in you. Demonstrating how badly informed you are can have the opposite effect, especially if you have been conducting your evaluation for some time.

You may find that some interviewees will ask whether their interviews will be confidential. Now it should be clear that if you are to act as an information broker you will be indicating to some colleagues what others are saying. Nevertheless, you can assure colleagues that they will not be quoted directly without their permission and to this extent interviews are confidential. For most of your colleagues it is fairly easy to disguise who is saying what. It is more problematical when it comes to the headteacher or a member of staff who is particularly central or observable as far as an issue is concerned. In this case you cannot offer confidentiality, merely tact. It can be a help if a high profile figure is prepared to raise important questions and thereby encourage others to respond. This could assist in creating the debate that is so critical to evaluation. But be careful that the debate does not become an argument which rebounds on you, and you are seen as the instigator. There is a fine line to be walked here and it is sometimes very interesting for the stakes to be raised a little.

It is often helpful to have a steering committee or group. This will give you some protection and credibility but could also interfere with your work. A strong character, for example, could undermine your work by demanding measurement and certainty. Select people carefully and do not include people who you cannot trust reasonably well but also look for those who have power and who approve of your endeavour [as opposed to your methods and conclusions].

Keep a diary of events. This will almost certainly become more than just a record of what has happened and when. The record element is useful enough, especially if you are questioned. You must know what happened and in what order, i.e. when. In addition, most diaries become reflective and analytical. A good diary will enable assist you to analyse as you go along and then examine your own reasoning when you come to write a final report.

Ultimately the most important part of this stage is attempting to clarify what has happened and what the participants feel about these events. In this sense evaluation is not a very glamorous activity. It requires enormous attention to detail and a persistence aimed at getting things right. You will find that some interviewees will not have the clear understanding that you require and others will not be very keen to explain or reveal their feelings, particularly if events are not very flattering to them. It is often after an interview has been completed that the important questions and inconsistencies begin to emerge. You will almost certainly have to return to some interviewees. The picture will be made more complex still when you consider that your colleagues will interpret situations and actions quite differently.

E. WRITING UP AND REPORTING

This is usually the most arduous part of your evaluation. It is at this stage that you have to sift out the central issues, establish any connections and supply the appropriate evidence. Your final report should be readable and anticipated. There should be no shocks at this stage.

One way of smoothing the path is to provide interim papers and/or talks. These will raise questions for participants and have the added benefit of contributing to the debate that needs to be part of your evaluation. All reports, including the interim ones should be to the point, concise and written in language that the participants are familiar with. Long reports tend to be put in draws and not read and if they are not very congratulatory there is a tendency to rubbish them if they are tricky to read. You may find it helpful to talk to a report at an arranged meeting and then record any responses as part of your evaluation.

The final report will have to be more polished than earlier ones. Final reports are often needed to feed into policy debates or for funding purposes and must 'look the part'. It should also be borne in mind that the final report maybe the only document by which the evaluation, and the programme it evaluates, will be judged and/or remembered. It is often difficult to know when to stop gathering data as important matters occur at a late stage during, or even after data, is collected. Be prepared to add in vital points or change some comments in the light of these events.

Even if colleagues are prepared, there is almost certainly going to be criticism of a negative report or of negative points. You will be relieved at this point if you have collected your data carefully, acted ethically, triangulated your data and drawn conclusions thoughtfully. In this respect it is important that you have recorded positive responses and it is a better idea to place these at the beginning of your report to be followed by the negatives if there are any. You should also give plenty of room to changes which have occurred in the light of criticism or reflection. It can be helpful to have a summary document for discussion purposes.

You should be clear as to the central issues which have arisen and which need to be discussed. You could use a metaphor which captures a major point for discussion but do not overdo it. The writing has to be clear and unambiguous if it is to serve the decision making process. Anything too dramatic in the writing can take over as an end in itself and distort a meeting so that the writing becomes more important than the issues themselves.

A major consideration of the evaluator at this stage is the effect of her report. Inherent in the notion of democratic evaluation is that the evaluator is not acting in her own right but on behalf of the participants. There is a long history of administrators and officials handing down edicts from on high only to find that in practice they are ignored and replaced by the concerns of practitioners on the ground. The evaluators task is to articulate the views of practitioners, not to make recommendations based on her own perceptions. It maybe possible to draw some simple lessons that all stakeholders agree with. These can be pointed out and probably decisions will be made. In other cases the evaluator can only report differences of opinion which will then have to be discussed and decided upon.

In the latter case the evaluator should not be simply swung by numbers. That is to say that even if most stakeholders believe that one decision should be made and a small group believe in another, it should not be concluded by the evaluator that the majority should hold sway. If this matter has not been sorted out beforehand then participants should have the opportunity to discuss both possibilities while preparing to make a decision.

The moral is that change should be rooted in the views of stakeholders and not 'pushed' by the evaluator or the management. The latter could make matters difficult, especially for the insider evaluator. There is no stronger defence than a carefully presented case based on good data. Yet you should be warned that this may not be enough. If the conclusions of your evaluation are ignored you should at least be able to defend your methodology and practice as an evaluator.




Return to contents page.