Introduction
π΅π»ββοΈ As a person performing reviews, we suggest you read the pages: Add feedbackο»Ώ, Accept/ reject tasksο»Ώ and Read feedbackο»Ώ.
π¨πΌβπ¨ As an annotator, we suggest you read the pages: Read feedbackο»Ώ and Act on the feedbackο»Ώ.
In-app feedback aims to enable clear and efficient communication between the reviewing person (giving feedback) and the person who did the initial task, respectively, who receives the reviewed task (receiving feedback, acting on it, responding to it). Feedback passed around outside the app via Slack, Teams, email, or spreadsheets is slower to act on and prone to errors.
- Give feedback directly inside the app rather than via spreadsheets or other ways
- Give feedback with direct reference to the annotated object in an image or a location inside a point cloud
- Give feedback inside the normal workflow with less delay after task completion
- Make small corrections inside the tool to avoid another iteration for small errors
If enabled, the "Review Phase" starts once a preliminary annotation exists, and its flow is as follows.
- (An Annotator annotated a task and potentially already did some corrections)
- A Reviewer reviews the task:
- Potentially adds Feedback and/or Correction Requests and
- Most importantly decides whether to Accept or Reject the task as a good final annotation in accordance with the guideline.
- An accepted task is done and soon ready to download by the client.
- In the future Annotators will also receive feedback from Accepted tasks, today that's not supported yet.
- A rejected annotation is assigned to an Annotator, if possible the original one, to:
- Correct the Correction Requests and
- Acknowledge or act on the Feedback
- Reply to both of the above, if needed
- Submits the corrected annotation
- A reviewer reviews again and she still Rejects step 4 is repeated until she Accepts.
In our review phase we use two types of feedback: Correction requests and Advice.
A correction request represents something the person reviewing a task want to have corrected. This means if an annotator recieves a correction task with correction requests in it - she is expected to correct the error the reviewer have indicated.
These items represents something the person reviewing wants the annotator to know or think about, but not something they expect you to correct. Advice could be things such as encouraging words and general tips on how to improve.
Discussing on the right way to do annotations can quickly become complex and details start to matter. Your feedback conversations might be much better if you consider the following tipps.
- Concentrate on the annotation content, not the person
- For example, "you forgot to add a box here" vs. "I think that according to the guidelines there should be a box for this parked car"
- Also give positive, encouraging feedback
- Try to be specific
- For example "behind the trees there's some parking cars missing" vs. "To me it seems there's five parking cars behind the trees"
- Feedback is your personal opinion and might be debateable, phrase it as such
- Be aware of responses
- Be open to different opinions
- Ask for clarifications if you don't understand how to act on feedback
ο»Ώ
