Exam analysis report

The exam analysis report provides detailed test-level insights for both written and OSCE exams, and is designed to assist in determining how the exam has performed overall.

Running the exam analysis report

To run the exam analysis report:

• Step 1 of 4

  Navigate to your exam then Set standard and View reports.

  

• Step 2 of 4

  Click Create new report and Exam analysis report.

  

• Step 3 of 4

  Select the date for which you wish to run the report. You can also optionally give the report a name and select any blueprint category to narrow the results to include only items tagged to those values.

  

  Click Save to commit.

• Step 4 of 4

  Click Preview to view the output.

  

  You can also export a copy of this report to PDF.

Report output 

The report output is shown below:

• Metric	Description
  Number of candidates	The number of candidates present in the exam
  Number of items	The number of items present in the exam
  Minimum score	The score achieved by the highest scoring candidate in the exam
  Maximum score	The score achieved by the lowest scoring candidate in the exam
  Median	The score value situated in the middle of the sorted list of scores (i.e. the point at which 50% of scores fall above and below).
  Mode	The most frequently occurring score value. If there is more than one value the smallest is returned. If there are no most common values the smallest score in the exam is returned.
  Mean	The simple average score for the cohort (calculated as the sum of test scores divided by the number of test scores).
  Standard error of the mean	A statistical estimate of the error in the sample mean (calculated as the standard deviation divided by the square root of the number of data points). As the number of observations increases the standard error value decreases.
  Standard deviation	A representation of the dispersion of scores relative to the mean (calculated as the square root of the variance). A higher value means the scores are more spread out, whereas a lower value means they are more tightly clustered.
  Skew	Indicates whether data is normally distributed. A symmetrical distribution will have a value of 0. A value >0 means the data is bunched to the left with a longer right tail (positive skew). The inverse is true for values <0. https://www.spcforexcel.com/knowledge/basic-statistics/are-skewness-and-kurtosis-useful-statistics#kurtosis
  Kurtosis	Indicates the presence of outliers in the distribution. Values >0 have heavier tails and values <0 have lighter tails. https://www.spcforexcel.com/knowledge/basic-statistics/are-skewness-and-kurtosis-useful-statistics#kurtosis

• Metric	Description
  Cut score	The test passing score, as determined by the standard-setting methodology used.
  Cronbach's alpha	Cronbach’s alpha is a measure used to assess the reliability, or internal consistency, of a set of scale or test items. In other words, the reliability of any given measurement refers to the extent to which it is a consistent measure of a concept, and Cronbach’s alpha is one way of measuring the strength of that consistency. Cronbach’s alpha is computed by correlating the score for each scale item with the total score for each observation (usually individual survey respondents or test takers), and then comparing that to the variance for all individual item scores. The resulting α coefficient of reliability ranges from 0 to 1 in providing this overall assessment of a measure’s reliability. If all of the scale items are entirely independent from one another (i.e., are not correlated or share no covariance), then α = 0; and, if all of the items have high covariances, then α will approach 1 as the number of items in the scale approaches infinity. In other words, the higher the α coefficient, the more the items have shared covariance and probably measure the same underlying concept.
  SE of measurement	The Standard Error of Measurement (not to be confused with the Standard Error of the Mean) gives an indication of the spread of the measurement errors, when estimating candidates' true scores from the observed scores. It is calculated from the reliability coefficient (Practique uses Cronbach's alpha). It is assumed that the sampling errors are normally distributed. The SEM is calculated as: SEM = S(1 – rxx)0.5 where S is the standard deviation of the exam, and rxx is the reliability coefficient (Cronbach's alpha). The key application of SEM in risr/ assess is to apply a confidence interval to the cut score. For example, if you would like to be 68% sure of the pass/fail decision, the SEM indicates that the candidates within 1 SEM of the cut score may fluctuate to the other side of the cut score should they take the exam again. For example, if you wanted to be 95% sure of your decision on outcomes, an SEM multiplier of 1.96 can be applied. These figures are based on the Normal Distribution. Assess applies this on the positive side for most Standard Setting methods, as we are dealing with competency exams. In practice, what this means is that you are 95% certain that the passing candidates’ scores represent their true scores.
  SEm multiplier	The SEm multiplier used during standard setting
  Error (SEm * multiplier)	The difference between the cut score and the cut score with an applied confidence interval (calculated as the SEm x the SEm multiplier)
  Pass score rounded	The confirmed passing score with any rounding applied
  Pass rate	The number of students who, based on the confirmed passing score, have passed the exam

• 

  The cumulative percentage chart shows the proportion of the cohort accounted for as the total test score increases. Test scores are mapped to the x-axis with the cumulative proportion of candidates accounted for at each scoring increment on the y-axis.

  In the example shown the scores range from ~44% to ~81% and increase reasonably steadily between those values. The chart can be useful to determine whether the full scoring range is being used and whether scores are concentrated around any particular point.

• 

  The items analysis chart cross tabulates the facility (difficulty) score (calculated as the sum of candidate scores / the sum of maximum possible scores) and discrimination values to show comparative performance values for each item.

  Hover over each point on the chart to display the underlying facility and discrimination indeces.

• 

  A cumulative frequency table showing how many candidates would pass and fail if a threshold for a minimum number of items to pass were set. Commonly used in OSCEs where candidates are required to pass a minimum number of stations to avoid cross compensation of poor and excellent performances.

  in the example shown, if you were to set the minimum number of stations to pass at 4, then 2 candidates would fail the exam.

• 

  A table showing the number of candidates that would pass and fail the exam based on the cut score + several commonly used confidence intervals.
  In the last row, the actual SEm multiplier is shown as well as the resulting outcomes.

• 

  This chart plots the candidate’s total percentage score for the exam against the number of individual stations passed. The plot also shows the intersection between the cut score (with the cut score for several commonly used confidence intervals) and any minimum station pass requirement set.