OSCE scoring elements

The individual scoring instruments in an OSCE mark sheet comprise three different elements; schema, criteria and sections. Each is described in the table below.

Element	Description	Setup
Schema	This is the scale on which the criterion is being measured (e.g. 0/1, 0/1/2/3/4, pass/fail etc.). Essentially the options available from which the examiner can select.	Can be set up globally or individually per mark sheet row.
Criterion	The criterion is the conceptual entity being measured. This could be a checklist item (e.g. "wash hands") or a scoring domain (e.g. "communication skills").	Can be set up globally or individually per mark sheet row.
Section	This is the part of the mark sheet in which each criterion field is grouped and contained. For example you may have "checklist" section and "feedback" section to aid presentation and organisation of the mark sheet.	Always set up at the global site level.

Global vs local criteria

Whilst it is possible to build mark sheets almost entirely per individual item (except sections, which always need to be set up at the site level), doing so would be an almost impossible task for most users. This would mean every mark sheet criterion would need to be built manually and every schema (including each point on the scale) would need to constructed manually per mark sheet row.

To make things more efficient we can use a combination of global schema and global criteria to accomplish large parts of this task.

The term global criterion in this context should not be confused with global mark, or similar, which is sometimes used to denote the overall judgement provided on an individual mark sheet. In this context, we mean any mark sheet criterion that can be set up at the site level and re-used across items. That could be a component mark OR an overall judgement.

When a criterion is set up at the global (site) level it can be applied as appropriate throughout the site without having to be rebuilt in each individual mark sheet.

Common applications for this could include:

• An overall judgement criterion that appears consistently across mark sheets. 
• Consistent scoring domains that are used across mark sheets. 

When a global criterion is created, it will be automatically applied to ALL items in the site by default. To control where the criterion appears, the ApplyIf function should be used.

Checklist vs domain scoring

There are generally two broad methods of scoring OSCE stations; checklist style and domain-based scoring. Using the former method, candidates are expected to perform the task and are scored on a prescriptive set of actions observed by the examiner. Domain-style scoring uses broader (and often consistent) measurement categories across the course of the exam and can be useful for more holistic consultation style examinations.

Either method may or may not be supplemented by a global, overall judgement or standard-setting mark as appropriate (especially where regression-based standard-setting methods are used).

Both of these scoring methodologies can be accommodated in risr/ assess.

The mechanics of the setup for both are largely similar. The main difference being that a checklist would tend to use a 0/1 or 0/1/2 schema (and there would be more of them) whereas domain-based scoring would normally use fewer criteria with a wider, or sometimes descriptive measurement scale (e.g. Poor/Borderline/Good/Very good/Excellent).

The other thing to consider is that where checklist marking is used, the criteria will almost always need to be built locally per item because the checklist is unique to the item. Where consistent domains are used, efficiencies can be achieved by setting the criteria up at the global site level and using the ApplyIf function to automatically construct mark sheets.