Combining exams

risr/ assess offers the possibility to combine the raw data from two separately administered exams into a single entity ready for result processing and standard setting. 

There are many such potential uses for this. Perhaps your exam is administered over several days or over several sites. Running these as separate exams in the assess system can offer greater control over how they are delivered locally, but the data collected still needs to be processed as a single statistical event. Alternatively, you might run a programmatic assessment approach whereby smaller exams are conducted throughout a year of study, but summative outcomes are calculated using the combined dataset. 

The combining function can help you to achieve all of these things.

When combining two or more exam events, one of two things must be true about the component exams. Either:

1. The candidates are the same in each exam but the items are different.
2. The items (including the item versions) are the same in all exams, but the candidates are different.

The system also supports sequential or exemption-style assessment methodologies whereby a subset of the cohort is required to sit more items so that more accurate judgments about their overall competence can be reached.

Whilst raw data can be combined to process examination outcomes, as an examination management system, it is not the purpose of risr/ assess to combine and maintain separately processed outcomes as might be found in a virtual learning environment or candidate records system.

Combining the exams

In the following example, we will combine two exams with the same set of candidates, but a different set of items.

To combine the exams:

• Step 1 of 5

  Navigate to Exams then Create combined exam.

  

• Step 2 of 5

  From the next screen, you can give your combined exam a title.

  

  Select whether to specify a date for the exam or to use an inherited date from the component exam. Also select the exam type, noting that only exams of the same type (written or OSCE) can be combined.

• Step 3 of 5

  Now select the component exams that you wish to combine. All eligible exams will be presented but can be filtered using the search box. Click Add next to each one you would like to include. 

  

  Next select the combination method.

  1. Same candidates and different items
  2. Same items, different candidates
  3. Sequential examination

  Click Verify.

• Step 4 of 5

  The verification process will run an operation to determine whether the data in the component parts can be combined.

  

  Subject to successful verification, click Confirm and create.

  The mark combined exams as hidden option hides the component exams from the exam list and excludes the data from any aggregate item performance. This is usually advisable so that data is not double counted (i.e. once in the component exam and once in the combined one).

• Step 5 of 5

  The newly combined exam is now available in the list of exams.

  

  There is a banner explaining that this is a combined exam with links back to the original component exams.

  You can now proceed to the standard setting for the combined set of data.

  The error message will almost always be present. This is nothing to worry about and is simply because the system recognises that the combined exam cannot be organised into a circuit structure.

If you have normalised your items via the 'set standard' page of the exams you are combining, please note that scores will be reset to the pre-normalised raw values in your combined exam. If you want the normalised values to remain in place when creating a combined exam, you will need to set the normalisation at an item set level rather than at an exam level.

Sequential assessment

Sequential, or exemption, style assessment is a specific methodology for assessing candidate competence. 

Generally, the full cohort will be required to sit the first part of the exam (let's say 10 stations). When the results are processed, a confidence interval is added to the cut score and any candidates who have achieved higher than that score are deemed to have passed the exam (or become exempt from sitting the remaining stations). Candidates who have not achieved the score are invited back to sit a second part of the exam so that more data can be collected on which to base a decision.

Given the resource-intensive nature (particularly of OSCE exams) this can achieve efficiencies because those candidates who have reached the required level of competence need only sit a smaller number of items. Those about whom there is less certainty (i.e. borderline or failing candidates) can be invited back to sit more items so that a more reliable decision can eventually be reached about their level of competence.

The process for combining the exams is almost identical to that described in the previous section, only with some important caveats. 

• At the point of combination select the Sequential exam option as the combination method.
• The order of the exams in the list now matters. The first exam in the list must be the first exam with the greatest number of candidates. The second exam is a supplementary one with a smaller number of candidates.

When the combining task is processed, the successful candidates from the first exam are excluded from the standard setting activity in the combined exam. The scores for candidates who have sat both parts of the exam are combined and can be standard set as appropriate.