Very short answer question (VSAQ)

The very short answer question (VSAQ) requires test-takers to respond to a prompt with very short answers. The response box is open-ended (i.e. there are no answer options from which to select), and respondents should be asked to provide responses of usually no more than 2-3 words and on a single line of text.

The VSAQ item supports a mixture of automatic and manual scoring. Matching responses are grouped together and can be evaluated in bulk. This is described further in this article.

Building a VSAQ item

To build a VSAQ item:

• Step 1 of 3

  Navigate to Items and Create new item.

  

• Step 2 of 3

  Complete the fields on the new item screen.

  

  1. Item type: Select Short answer question from the dropdown menu
  2. Item title: Give the item a name. This is entirely free-text and at your discretion, but it is required. This is not visible to candidates during the exam, but it can optionally be used later in the candidate feedback report.
  3. Item number: The item number auto populates but can be overwritten. Each item type maintains its own sequential index from 1-n (e.g. ARQ1, ARQ2 / SBA1, SBA2 etc.). By default this field will receive the next available sequential number for that item type.

• Step 3 of 3

  The basic form fields for the item type are now made available.

  

  1. Item number: This is the item number established in the previous step. It can no longer be changed. 
  2. Question author(s): Unless specified, the question author will default to the current user. Alternatively you can select another author if you are inputting the material on someone else's behalf. Start typing a name in the field and the system will suggest names based on the list of users present in the system.
  3. Item title: The item title entered in the previous step. This can be changed here.
  4. Summary: This is the item stem. You can use the WYSIWYG editor to write your question prompts. 
  5. Comments: The comments field is used for notes against the item. These do not appear to candidates and are for administrative use only.
  6. Item notes: The item notes field can be made available to candidates in the candidate feedback report and is designed to be used for any predetermined answer justification.
  7. Levenshtein threshhold: If the test-taker provides a response that is not one of the given ones, but is within the Levenshtein threshold (i.e. the minimum number of character changes to achieve an identical string), no credit will be given, but the response will be marked in amber so a human marker can evaluate.
  8. Angoff: An Angoff rating for later standard setting can be banked against the item. If a value is given, this will be used if Angoff is selected as the standard-setting method.
  9. Difficulty and relevance: If Ebel standard setting is used, you can bank the difficulty and relevance values agains the item.
  10. Response to this question is mandatory: select if you want a response to be mandatory or not.
  11. Response to this question is case sensitive: select if you want the question to be case sensitive or not. 

  Click Save item to commit the changes.

Adding Responses

Select the Add new responses button and complete the response name in the free text box.

Mark: This is the number of marks a candidate will receive for answering correctly. A correct answer would typically have a value of 1.0, with partially correct answers given a score relative to this.

Delete: Select this box to delete a response you have added.

Item preview

In the browser application, the VSAQ item appears with the question on the left. Test-takers are required to complete their answer in the free text box on the right.

Dealing with new recognised responses

As described in the VSAQ marking article, it is possible for markers to transform responses given by candidates during an exam to new entries in the list of recognised responses in the item. When this happens a new version of the item is automatically created with the new recognised answers present in the list. This is shown in the below image.

Note how the Magenta response discussed in the marking article has now been added to the list.

Before the item is next used the author can review the list of responses and must then re-submit the item for approval. Once approved it can be used in future exams with the new list of recognised responses.

Over time, and as the list of recognised responses builds up, the marking burden becomes less onerous.