Exam Grading
This tab is only available if you have been added as an examinator or supervisor for one or multiple annotation requests.
If you have been added as an examinator or supervisor for any request you will be able to access the exam page and the tab Exam grading.
When opening this tab you will see a list of annotation requests containing exams to grade. You can also access exams you have started grading but not finished here. Below we will describe how to find and claim an exam for grading.
The exam grading functionality is mainly intended for examiners and therefore mentioned a lot, but a supervisor also has access to claim and grade exams just like an examiner.
Within each annotation request the team of examiners together grade a group of exams one by one. Each exam in the group is graded by one examiner, the examiner which claimed the exam.
You claim an exam from the list "Exams to grade". You can only claim one exam at the time - meaning you have to grade your claimed exam to be able to pick up a new one. You can also be removed as an examiner by your manager, leading to the exam being available for other examiners to claim instead.
When claiming the exam you will open it in its grading view.
Number of exams left to grade This number show how many unclaimed exams that are available for grading within an annotation request. You and your examiner collegues are together responsible for grading all of them.
You can find your already claimed exam at the top of the page in the section "Your ongoing grading" until you have graded it. This means that you can pause your grading and continue at a later point if you want to.
You grade the exam after either claiming it or resuming it from the exam grading tab. In the grading view you will access the tasks that was part of the exam as well as statistics about them - helping you reason about the grade.
In each exam the examined user have conducted one or multiple exam tasks. By looking at these tasks and the provided statistics for them you will decide if the examined user is ready for production tasks or not.
You can open each of the tasks in view mode by clicking the "view task"-button.
For each of the tasks you can find statistics:
- Task completed - The date and time when the task was submitted.
- Total task time - The time it took to completed the full task.
- Number of objects - The number of objects in the task.
- Average time per object - The average time spent per object in the task.
An object is a unique object in the world which can be annotated in multiple sensors or multiple frames. An object has a unique object id, which is visible in the annotation tool's object list. Read more.
We strongly encourage you to write down feedback that you later can share with the person that did the exam and its tasks. This so they get to learn about the strenght and weaknesses, and also understand why the either passed or failed the exam.
The feedback can be shared in the app. Upload your feedback as a PDF file.
When you have come to an conclusion if the examined user is ready to work in production tasks or not - you need to select and submit the grade. To help you decide this you should have been given quality and speed requirements by your manager.
Pass The examined user fulfilled the quality and speed expectations of the exam. The examined user is ready to work with production tasks related to the exam.
The consequences of passing the user is:
- If the user did their exam in the same annotation request as production tasks are being done in - they will automatically be given production tasks (if there are any available).
- If the user did their exam in a annotation request created specifically for onboarding - they will have to be added as team members in the relevant production request. Who adds the user to the production request will vary, so if you are unsure double-check with your manager.
Fail The user didnβt fulfill the quality and speed expectations of the exam. The user needs more training before they work with production tasks related to the exam.
The consequences of failing the user is:
- When you fail a user they will automatically start a new exam - configured as the previous one but with new tasks.
When you have graded the exam you will be able to go back to the exam grading page and claim a new exam. Before doing that make sure to send feedback to the examined user, or at least collect the user name and email so you can send it later.
