EUTEACH: CONCEPTS AND TOOLS FOR THE EVALUATION OF THE TRAINEES

Introduction

There are several methods which can be used by the teacher (or another person in charge of the assessment of the impact of the course/session) to evaluate the trainees/learners who take part in courses based on the EuTEACH curriculum. The choice of evaluation method will depend on the approaches which have been used during the course. This sheet outlines some of the concepts and tools involved in the evaluation process of the learners. The EuTEACH curriculum focuses on the acquisition of knowledge, attitudes and skills which should enable practising physician to improve their approach when helping their adolescents patients and also with the quality of the curative and preventive care that they provide.

The main evaluation methods presented put various levels of emphasis on each of the three components outlined above. If the audience is made up of paediatricians, the teacher may elect to put less emphasis on the knowledge of growth and puberty and highlight the importance of attitudes and skills in the adolescent-physician encounter through interactive counselling techniques, while for psychiatrists, the emphasis may be on the understanding of physical growth and pubertal stages. Thus, it is essential that the teacher designing and managing the course sets clear objectives (see the planning of the session document) according to the needs assessment of the targeted audience, and that the evaluation methods are then chosen in the light of the respective weight of knowledge, attitudinal and skills used as components of the program. Thus, to a large extent, the choice of the evaluation tools when planning a course will help the teacher to select specific and measurable objectives (and vice & versa !). 

The teacher has also to decide how he will rate the evaluation; in other words, to decide at what level he considers that the learners has attained the selected objectives. While most of pre-graduate training courses use pass/fail approaches (summative evaluation), continuous medical education tends to use a more formative procedure. That is, that learners are requested to identify areas in which they desire to improve and then they are given credits once they have reached a pre-defined level of performance (whether in the field of knowledge, attitudes or skills). The certification can be given once a certain amount of credits have been collected by the learner. One interesting way to approach the assessment of course graduates is to distinguish between four types of testing depending on the learner’s ability to :

• know (pure theoretical knowledge)
• know how ( in which circumstances to use specific knowledge or adopt specific attitudes)
• show how (demonstrate the ability to use knowledge and skills or adopt specific attitudes)
• do (demonstrate in his everyday work that he has integrated the objectives of the course)

While the first three steps are usually assessed after a defined session (either immediately after the session or after several weeks/months have passed), the teacher may elect to assess the impact of his curriculum by directly evaluating the actual practice of the physicians who took part in the course ( the «do» component). Although this approach is time-consuming and difficult to implement, it is ultimately the most effective way to evaluate as to whether the participation in the curriculum has really brought significant changes in the quality of adolescent care.

Evaluation tools

The main evaluation tools are displayed in the next table. Depending on whether the part of the course which is evaluated focuses more on knowledge or more on attitudes or skills, the person developing the evaluation of the course (which may not always be the person designing the content of the course) can choose from amongst the specific approaches as outlined in the next table. Obviously, the availability of specific tools (videos, simulated patient) as well as the timeframe will influence their decision.

A. EVALUATION OF THE TRAINEE

A1. The multiple-choice questionnaire

This is one of the most commonly used methods to evaluate improvement in the learner’ knowledge. However, in many instances, it only assesses pure knowledge. The next method presents more elaborate questions with several items linked together, which helps address the issue of knowledge utilisation («know how»).

A2. The application test

This incorporates a variety of multiple-choice questions and thus assesses higher levels of thinking and the capacity to synthesise knowledge.

A3. essay and other written reports

Students are asked to write a text of a given length (summary, essay, portofolio, etc) presenting a synthesis of a specific subject. This may include linking certain subjects of the course with clinical examples; addressing a certain theme within a given population; compiling and presenting the literature in a given area; or synthesising the content of a module in the form of a ’report on its relevance’ for their own practice, etc.

A4. Oral interviewing

This is a convenient but time-consuming way of evaluating higher levels of knowledge, knowledge integration and application of the knowledge in current practice. It also allows one to make an estimation of attitudes and skills. It is a flexible method but one which may be biased by the subjective biases of the examiners !

A5. Group interview

This method can be used to assess both knowledge and attitudes, as well as other capacities in term of interactions and adaptability.

A6. Direct observation of patient-physician encounter

Direct observation allows for an assessment of integrated knowledge, attitudes and skills but the judgement may be biased by subjective interpretation by the observer.

A7. Objective structured clinical examination

An examination which consists of multiple short sections, usually five to ten minutes, in which the learner is asked to perform specific well-defined tasks of a clinical nature. The advantage of this approach is that it focuses on the « know » and « know how » components thus allowing assessment of variety of items in a brief period of time in a pre-determined structured way which diminishes the risk of interpretation biases (as each learner encounters the same questions and tasks).

A8. Videoing AND reviewing the video

The use of videotaping a clinical examination is that it allows a more ‘in-depth’ review of the encounter’s process and content and therefore is a more objective assessment tool than the preceding one.

A9. use of standardised or simulated patient

An adaptation of the above two approaches is the used of a ‘standardised’ or ‘simulated’ patient, who has been trained to be behave in a certain way and thus allow for a more standardised estimation of the content and process of the encounter.

B. EVALUATION OF TRAINEE’S PRACTICE

B1. Self perception

The ’learners’ are asked to report on the way their approaches and current practice has been changed by their participation in the course. If the account is not plotted against other more objective measures, it represent a simple approach, but of questionable validity.

B2. The chart review

This method allows for an indirect assessment of both the ‘relevance’ and ‘comprehensiveness’ of the investigation and, to some extent, of the ‘accuracy’ and ‘validity’ of the diagnoses and the quality of care. It is a time-consuming task, because in most cases, such an appraisal requires the intervention of several experts with the added need for inter-reliability scoring.

B3. Observation

As stated earlier, (A6-A9), this is one of the most powerful tools for assessment (video, simulated patient, etc.).

Designing an evaluation

We suggest the following procedure when designing the curriculum objectives, content and evaluation :

1. Decide whether to put the emphasis on the summative or formative aspects of the evaluation.
   
2. Select, for each of the stated objectives of the course what you consider to be the most appropriate evaluation method.
   
3. Decide, when designing each part of the evaluation, on which of the various components of knowledge, attitudes and skills you want to put most emphasis, and whether you want to assess their ‘pure’ acquisition (i.e. «  know » and «know how » components) or if you want to demonstrate the learner’s ability to implement what he/she has learned into practice («show how » and «do» components). 
   
4. Create a draft outline of the evaluative instruments you are going to use, and if possible pre-test it.
   
5. Verify the time the testing process will require.
   
6. Determine the scoring methods that you will be using.

Short bibliography

1. Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum development for medical education. John Hopkins University Press, Baltimore, 1998 
   
2. University of New Mexico School of Medicine. « State of the Art » Assessment Manual. Blueprint. 
   
3. Green LW, Lewis FM. Measurement and Evaluation in Health Education and Health Promotion. Mayfield Public, Palo Alto, 1986 
   
4. Guilbert JJ. Educational handbook for health personnel. WHO cat Nb. 1898, Geneva, 1998