Mark sheet global criteria

Marking criteria are basic building blocks to scoring structured assessments. risr/ assess allows you to create scoring criteria at the global/site level so that they can be used in two different items types without having to create new ones every time.

• OSCE item mark sheets
• Multipart Written Question (MWQ) with Short Answer Question (SAQ) subpart

Global criteria will apply to these items created in the site unless controlled using the ApplyIf function to specify where they should appear. This saves time for authors and ensures consistency and data integrity.

The term "global criterion" in this case should not be confused with a "global mark" (or similar). Commonly, a "global mark" might be requested to collect an examiner's overall judgement of the candidate's performance and is then used in setting the passing standard for the exam. In this context, a "global criterion" is simply one which is built at the site level.

You can set up any scoring instrument you intend to use on mark sheets at the site level and apply it to individual items. This is most useful when using consistent scoring domains across stations (for example "History Taking" or "Examination Skills"). For OSCEs where checklist-style scoring, steps are usually specific to each station, so setting them up globally might not be efficient unless some steps are common across stations.

Building a global criterion 

The process of building a global criterion is largely similar between an OSCE mark sheet and an MWQ item with SAQ subpart(s), with only slight nuances in implementation. The following outlines how this is achieved with an OSCE marksheet. The subsequent section outlines how this is different for a written global criterion. To enable the various standard setting options please reach out via the service desk.  A full list of current methods can be found on the Introduction to Standard Setting page.

Global Criterion - OSCE Marksheet

• Step 1 of 8

  Navigate to Settings > Global Criteria

  

• Step 2 of 8

  Click Add new criteria

  

• Step 3 of 8

  In the modal box select the mark sheet section you want the criterion to appear in.

  

• Step 4 of 8

  Type the label to appear against the criterion

  

• Step 5 of 8

  Select the Sheet type. This is the type of response field you wish to present to the examiner.

  

  • Discrete: A set of selectable responses (i.e. dropdown). Used where examiners are required to select the most appropriate value.
  • Boolean: A binary response field (i.e. checkbox). Used for checklist-style responses.
  • Free text: An open-ended text response. For example, enabling examiners to provide free text feedback.
  • Audio: Creates an audio recording response field. Examiners can record verbal feedback for the candidate.
  • Label: No response field is created. This acts as a text placeholder on the mark sheet and can be used for headings or explanatory text.

• Step 6 of 8

  Select the scoring schema to be used in the criterion

  

  The values available in the list are those set up under mark sheet schema. If Custom is selected a new schema will need to be built specifically for this criterion.

• Step 7 of 8

  Give the criterion a weighting.

  

  The weighting act as multiplier for the response provided by the examiner. So if, for example, the weighting was 2.0, the final value recorded would be the examiner's response (let's say 3.0) multiplied by 2.0 = 6.0.

  Common weightings that might be used here could be 1.0 if all mark sheet items are to be weighted equally, or 0.0 if the criterion is not intended to count towards the calculated total station score (e.g. where a global mark is being requested).

• Step 8 of 8

  Use the checkboxes to specify the behaviour of the criterion on the mark sheet.

  

  • Mandatory: Controls whether a response from the marker is required. If checked, the criterion is mandatory and the examiner cannot submit their mark sheet without completing the field (they will be prompted to complete the missing field). This is generally desirable as it helps ensure data integrity and completeness in the exam.   
  • Feedback visible to candidate: Controls whether the examiner's response to this criterion is visible to candidates in the Candidate Feedback Report.
  • This is a borderline marker: Designates the criterion as a borderline marker. If a borderline method is being used for standard-setting one criterion on each mark sheet must act as the borderline marker. This is the value against which all other data point will be regressed.

  Please note that the behaviour checkboxes are dynamic and will appear and disappear based on the value selected in the Sheet type field. For example, if Boolean is selected, instead of seeing the Mandatory option, you will see another option to make the criterion a critical stop point. If the Label option is selected, then all the behaviour checkboxes disappear.

Global Criterion - MWQ with SAQ subpart

• Step 1 of 6

  Once Borderline Group, Borderline Regression, Adjustable Borderline Regression or McManus Borderline Regression method are enabled on the instance, there will be a “Written Standard Setting” option listed under the drop down menu of Settings.

  

• Step 2 of 6

  Setting up the borderline marker is completed in a similar way to that of an OSCE.

  Create a schema or use a preexisting one from the “Manage schema” section in Settings. This example uses a preexisting schema “Borderline schema”.

  

• Step 3 of 6

  Select “Written Borderline Criteria” in Settings and click “Add new criteria”.

  

• Step 4 of 6

  A pop-up window appears where the appropriate schema can be chosen. In this instance, "Borderline schema" with the display text "Global Judgement" was selected.

  Clicking "Save" adds the criterion to the list of borderline criterion.

  

• Step 5 of 6

  After creating a new MWQ and adding a SAQ subpart, the borderline criterion will be added once the item is saved and the page refreshes.

  

• Step 6 of 6

  If a second Borderline Criterion is added to the Written Global Criteria, one will need to be excluded using the “Apply if” function, as only one global criterion can be used at a time. Only items tagged with the “Apply if” category will inherit the criterion.

  

Editing global criteria - OSCE Marksheet

Once the global criterion has been created users have several options available for editing

1. These are the values set during the previous steps now helpfully summarised in a table format. To adjust any of these use the Edit button and make the appropriate changes.

2. The CountAs function is used in conjunction with the site blueprint for data aggregation purposes. Essentially the data collected for this criterion can be "counted as" a specified value. There is a dedicated article about the use of the CountAs function here. Please note that this feature does not apply to written global observation criteria, as the global score is not included in the candidate feedback report.

3. The ApplyIf function is used to control where the criterion is displayed and on which mark sheets. In this example, it might be the case that you only want this criterion to appear on mark sheets where the station is tagged as "History taking". The dedicated article on ApplyIf is here.

4. The Admin button takes the user to the Criteria management tools page which can be used to push out any criteria changes made to items that are currently in an approved state. Another dedicated article on this function is available here.

5. The Remove button removes the criterion from the list and makes it unavailable to be used in future items being authored. It does not remove it from published items in which it is currently being used.

Criteria changes are strictly version-controlled at the item level. This is to prevent changes at the global level from cascading into active exams without a conscious decision being made to do so. Please refer to our articles on Global vs local criteria, Criteria management tools and Updating marksheet criteria for further information.