Usability testing policy
This policy covers all usability testing which takes place at the University of St Andrews and applies to all staff who are involved in any usability testing process. The policy applies to in-person and distance testing, as well as for testing on mobile or desktop devices.
Any School, department or unit within the University can request usability testing, but no usability testing may take place without the agreement of the digital communications team. The intention of this policy is to highlight the rules associated with organising, conducting and reporting on usability tests.
Organising and conducting usability testing in the University requires good communication between the digital communications team and the School, department or unit who wish testing to take place. This means there needs to be at least one key contact in the relevant department with whom digital communications can liaise with during the testing process.
Any change to the point of contact must be communicated to the digital communications team to ensure that the testing can continue or be rescheduled as necessary.
If a School, department or unit wants to test their web pages or an app, they first need to decide whether they want to conduct the testing themselves or have digital communications run the tests.
A range of audiences can be used to test a University website or app, such as:
- current students
- current members of staff
- prospective students
- members of the public.
However, you must not test:
- any member of staff who is involved in the project or web pages
- anyone who has seen the testing script before the test
- anyone who has undertaken the same test before.
Within reason, participants should know as little about the tasks and the theme of the test as possible before testing takes place.
Incentives are a way to encourage participants to test your website or app. Incentives are optional, and whether you need them will depend on the type of participant you require for testing. If an incentive is needed, this must be supplied by the School, department or unit who wish to undertake usability testing.
If you are testing a member of staff at the University, you must not provide a monetary-value incentive. This is because staff are already paid by the University, and if they are participating as a member of staff, the testing is an extension of their role.
Members of the public and current students can be given an incentive, and this can be determined by any budget you have access to in your School, department or unit. For example, in the past, teams have provided vouchers for students or gift bags filled with University merchandise.
A usability testing script must be created for usability testing as this ensures that all tests conducted in one round of testing are standardised, and thus the data is more accurate. When testing, the script must be followed exactly.
Digital communications will provide a template for the script. The team will also highlight how to create non-leading tasks for participants during the initial meeting and during the usability testing training session. It is the responsibility of the School, department or unit to create the first draft of the testing script, which must then be signed-off by digital communications.
Scripts must mirror the formatting in the script template provided by digital communications.
You should record individual usability tests when possible. By recording tests, you will be able to assess the usability issues at a later date and use recordings as evidence to back up findings in reports and meetings.
However, a participant must not be recorded without their written permission. This includes recording their visual reactions, actions, or their voice. To ensure you have the explicit permission of the participant, they must sign a recording consent form.
The recording consent form is a document owned by digital communications, and no other consent form should be used for usability testing. The form is provided to the School, department or unit who want to conduct their own testing prior to any tests taking place.
Storing the recording consent forms
If a School, department or unit wants to conduct their own testing, all recording consent forms must be returned to digital communications.
Upon completion of the testing, all recording consent forms must be labelled with the name and date of the testing, scanned as a PDF, and stored in OneDrive by digital communications. After all the forms are uploaded to the drive, the physical copies must be shredded.
Lookback is the only tool which should be used to record usability tests. For information about accessing Lookback to record tests, please see the Lookback access policy.
If members of staff wish to conduct their own testing, they must have completed the digital visa training course: ‘Usability testing: understand how your website works’.
In addition to the training, the key contact must have a suitable space in which the testing can take place. Ideally, this space will be quiet and have minimal distractions for test participants. If testing is conducted in-person, the location must take place on campus in a University building.
If testing is conducted through moderated remote testing, this must occur in a place which has a strong internet connection. However, the tester should not be connected to eduroam.
If part of a recorded test asks a participant log in to a system such as Outlook, MMS or Moodle, for example, the participant must be informed at the beginning of the test that they must log out of all systems at the end of the test.
Hardware used for testing
If conducting in-person testing, the digital communications team will provide the following hardware for testing:
- University laptop
The microphone must only be used to record the participant’s feedback if the participant has agreed to be recorded. The items above will be provided to the key contact when the School, department or unit is given access to Lookback.
A mouse will also be provided, as all participants should be given the option to use a mouse instead of the laptop's trackpad.
The level of involvement from digital communications in any usability testing at the University will largely depend on whether a School, department or unit wishes to conduct the usability testing themselves or if they require assistance.
Regardless of which testing process is chosen, the amount of time the digital communications team will spend with the School, department or unit will be decided during the initial meeting and will be documented in the follow-up summary email. This allocation of time is fixed and cannot be changed once the usability process has started.
When digital communications conduct testing
If the digital communications team is conducting testing on behalf of a School, department or unit, then digital communications will be responsible for reporting any findings. The School, department or unit may select the method that the findings are reported from a set of options.
An initial debriefing meeting with the main contact will be organised by digital communications shortly after the last testing session to discuss the main findings and confirm the next steps.
In addition to this meeting, staff may wish to also choose one from:
- a short report covering the main usability issues and some potential solutions
- a longer, more comprehensive report covering what the testing was for, what the usability outcomes were, recommendations and individual user feedback analysis.
You may also choose to have an in-person presentation to your School, department or unit covering what the usability outcomes were, any recommendations and snippets of user experience videos.
Digital communications aim to report back official findings to the key contact within a month of the final usability test. However, this may be subject to change based on the reporting options chosen by the School, Department or unit.
When a School, department or unit conducts testing
It will be the responsibility of the School, department or unit to report any usability findings to the relevant people. There are several options to consider for reporting, all of which are covered in greater detail in the training course.
During the reporting process, digital communications are happy to consult on any findings and advise on recommendations and next steps.
The findings of the usability tests may require a number of changes to be made. These changes may be ‘quick fixes’ but they also may be more complex and require significant developer or content editor resource.
If a School has changes that need to be made, the School’s computing officer should action those changes. If the School does not have a computing officer, they will need to send all small changes to firstname.lastname@example.org.
The same applies for units within the University. If units have access to developer resource, any reasonable changes should be put forward to them. If units do not have developer resource, they will need to contact email@example.com.
If the test results show that there needs to be a dramatic overhaul of a website, web or mobile app, then this will need to be raised with the University’s business transformation board.