OSCE Writing Guidelines

Guidelines for Developing OSCE Stations and the OSCE

An ‘Objective Structured Clinical Examination’ (OSCE) is an examination format for assessing students’ clinical skills.  An OSCE consists of a set of ‘stations’ through which students rotate.  A ‘station’ can be as short as 5 minutes and often they are as long as 20 minutes.  At each station there is an examiner who observers the students performance.  And most often there is a ‘standardized patient’ (SP) at each station, an individual who has been trained to play the role of a patient.  Recruiting appropriate SPs for an OSCE is important.  Criteria for selecting individuals to serve as SPs must be developed, to ensure consistency of SPs across testing centers.  First, the SP must be ‘authentic’ – that is, he or she must be the age, gender, weight, etc. of the patient described in the station.  In addition, the SP must be able to communicate in Indonesian.   Criteria may be developed relative a minimum level of education for SPs.  Staff working in the health professions schools or schoolteachers can serve as SPs.

 

At the time of a health professional’s graduation from school, OSCEs are used to assess history taking, physical examination, diagnostic and therapeutic and counseling skills.  There are times when an OSCE station also presents the student with written or oral questions related to the standardized patient encounter (e.g., what is your differential diagnosis of this patient, what would you prescribe for this patient).  Some OSCE stations test students’ procedural skills and may not use-standardized patients; rather they use models or manikins.   

Two of many resources available to guide NACEHealthPro the development and implementation of the ‘OSCEs’ to be used in the National Competency Examinations are the “Handbook On Competency-Based Assessment in Indonesian Medical Schools”, and the ‘Procedures Manual for the Medical Council of Canada Qualifying Examination Part II Test Centers’.  NACEHealthPro has copies of both these documents. 

The current plan for the National Competency-Based OSCE for each profession is to present 12 15-minute stations, with a minute between stations.  The total time required will be approximately 3.5 hours.  The first OSCE will be piloted for medicine and dentistry in 2011, with the real examination being introduced in 2012.   Most stations will use standardized patients; a few will use models or manikins.  

 

Below are a set of steps to follow in developing a National Competency-Based OSCE. 

Developing the General Plan for the OSCE

 

As initial steps in planning the National Competency-Based OSCE, the competencies to be tested in the OSCE and structure of the OSCE must be defined.

  1. Which competencies will be tested in the OSCE?  As discussed earlier in this paper, the general answer to this question is provided by the blueprint developed for the OSCE, as the blueprint will identify which competencies will be tested in statement of the National Competency-Based Standards for each health profession.  The competencies being assessed can include communication skills, history taking skills, physical examination skills, problem solving/diagnostic skills, patient management (counseling, treatment) skills, and professional behaviours.
  2. What will be the structure of the OSCE?  A decision has already been made to use 12 15-minute stations in each OSCE.  Relative to the blueprint for the OSCE, more detailed decisions on structure will need to be made on each station – e.g., will it use an SP or a model, will it present written or oral questions to the student?   Answering these questions will guide the development of a more detailed blueprint for your OSCE, and the development of  the individual OSCE stations

 

Developing an OSCE Station

 

In this section, the focus will be on developing an OSCE station that uses an SP.  Such stations are more challenging to develop than stations which use models. The logical steps in the development of an OSCE station are: 

  1. Define the purpose of the station (i.e., define the competencies that will be tested in the station) For example, a purpose of an OSCE station for an abdominal pain case could be to, “Demonstrate ability to perform an appropriate physical examination for abdominal pain secondary to appendicitis and be able to identify positive findings.”  The statement of the purpose of the station guides what to include and exclude in the case description and the scoring sheet. 
  2. Develop the case and instructions to the student.   The case should include patient’s name, age, setting (e.g., emergency, clinic, hospital ward), and relevant background information (if any).  A detailed case description must be prepared as a basis for training the SP to correctly portray the role of the patient, answer the questions posed by the students, and where possible simulate the responses when being examined.  The instruction to the student should specify what the students are asked to do in the station and the time limit.  For example, “A 16 years old woman has come to the Emergency Room with a 15-hour history of abdominal pain.  In the next ten minutes, conduct a focused and relevant physical examination.  As you proceed, explain what you are doing and describe your findings.” 
  3. Develop the Scoring Sheets for the Station
    1. Checklists which record whether something was done or not done are useful when assessing (a) key elements in data gathering, (b) procedural tasks involving specific steps, and (d) situations in which there is limited time available for training assessors and/or for marking.  If checklists are used, the list of items should be short (e.g., 5 or 6), and focus only on key elements – long checklists reward thoroughness of performance, and thoroughness is often inversely related to effectiveness.  For checklists you may wish to assign higher scoring weights to items that are more important.  In doing so it is important to keep the weighting system simple, and not get into debates about whether a weight should be a 2 or a 3.  The golden rule of weighting is ‘keep it simple’.  Weighting checklist items improves the face validity of your checklist, but weighting has been shown in many studies to not affect the reliability of the checklist scores. 
    2. Rating scales, which elicit judgments of the quality of a performance, are very useful on examinations used for certification purposes.  Rating scales, like checklist items, can be assigned different weights to reflect the relative importance of what is being assessed to the case.  Scores from rating scales generally prove to be more reliable than scores from checklists.  Most scoring sheets for OSCE stations present a mix of checklist and rating scale items.  Appendix C presents some sample formats for scoring sheets, and some sets of rating scales used for National Competency-Based examinations in Canada and Australia.   Additional sample scoring sheets are presented in the Handbook On Competency-Based Assessment in Indonesian Medical Schools, and are also presented on the Medical Council of Canada’s website, http://www.mcc.ca/en/exams/qe2/scoring.shtml.
    3. Scoring sheets can also present a section of questions that the examiner will ask the student in the last few minutes of the station.  To ensure 100% examiner consistency it is important that this section present not only the questions, but also their answers.  Examiners should read the questions to the student, and then simply check off if the student has or has not given the correct answers.
    4. The fourth component of an OSCE scoring sheet is a question at the bottom of the sheet which is used to set the standard or passing score for the station.  The question typically states, “What is your overall judgment of student’s level of performance on this station?”  Many different rating scales can be used to answer the question, such as

  4. i.      1 = fail, 2 = borderline, 3= meets expectations, 4 = exceeds expectations , or
    ii.      VU = very unacceptable, U = unacceptable, BU = borderline unacceptable, BA = borderline acceptable, A = acceptable, VA = very acceptable

    In the section below on scoring, it will be explained how these ratings are used to set the cutting score for the station.

  5. Scoring the OSCE
  1. Each station in the OSCE is scored by summing the numbers assigned to a student’s the correct responses as recorded on the scoring sheet.  The minimally acceptable score on the station (i.e., the cutting score) is defined by averaging the scores of the students who, on the last question on the scoring sheet, were judged to be ‘borderline’, or ‘borderline unacceptable’ and ‘borderline acceptable’
  2. The score on each station is usually presented as a percentage score, and the score on the OSCE as a whole is the average of the scores on each station.  In this way each station is given equal weight in the scoring.  The cutting score for the OSCE as a whole is simply the average of the cutting scores on each station.  This system of scoring is called a ‘compensatory scoring system’, because very good performance on some stations can ‘compensate’ for poor performance on others.  Many OSCE examinations also use a second scoring system, which simply counts the number of stations that are passed.  The ‘cutting score’ – ‘the minimum number of stations that must be passed’, is defined by a group of experts who review the stations comprising the OSCE, and reach a consensus (e.g., must pass 10 out of 12 stations to pass).  This latter system is called a ‘non-compensatory system of scoring’, since very good performance on some stations cannot compensate for poor performance on others.  Many National Competency-Based examinations employ both forms of standards – a student’s overall score must be above the cutting score for the examination, and the student must also have passed a defined number of stations. 

 

Other Issues in OSCE Development and Administration

  1. Just as with MCQs, OSCE stations should be pretested as part of the test development process, to refine the station and the scoring sheet, and to determine difficulty and discrimination indices. 
  2. OSCE stations should be stored and classified in a computerized item bank.  Again as for MCQs, the OSCE station classification system should be designed relative to the National Competency Standards for each health profession
  3. OSCE examiners should receive orientation and training, and should be selected to not have any conflicts of interest in their examiner role.  That is, ideally individuals who are not from their school or have been their teachers should examine students for example.
  4. OSCE orientation materials should be prepared for the students, to provide examples of OSCE stations, and scoring sheets.  Such materials can be effectively presented on websites, and can include videotaped samples of students in OSCE stations.  Examples of such orientation materials can be seen on the Medical Council of Canada’s website, (http://www.mcc.ca/en/exams/qe2/), or on the USMLE website (http://www.usmle.org/Examinations/practice_materials.html).