THE FEEDBACK TOOLS

Introduction

9min

πŸ•΅πŸ»β€β™€οΈ As a person performing reviews, we suggest you read the pages: Add feedbackο»Ώ, Accept/ reject tasksο»Ώ and Read feedbackο»Ώ.

πŸ‘¨πŸΌβ€πŸŽ¨ As an annotator, we suggest you read the pages: Read feedbackο»Ώ and Act on the feedbackο»Ώ.

ο»Ώ

Feedback inside the Review Phase

In-app feedback aims to enable clear and efficient communication between the reviewing person (giving feedback) and the person who did the initial task, respectively, who receives the reviewed task (receiving feedback, acting on it, responding to it). Feedback passed around outside the app via Slack, Teams, email, or spreadsheets is slower to act on and prone to errors.

Advantages of the review phase

  • Give feedback directly inside the app rather than via spreadsheets or other ways
  • Give feedback with direct reference to the annotated object in an image or a location inside a point cloud
  • Give feedback inside the normal workflow with less delay after task completion
  • Make small corrections inside the tool to avoid another iteration for small errors

How It Works

If enabled, the "Review Phase" starts once a preliminary annotation exists, and its flow is as follows.

Visualization of the Review Phase
Visualization of the Review Phase
ο»Ώ
  1. (An Annotator annotated a task and potentially already did some corrections)
  2. A Reviewer reviews the task:
    1. Potentially adds Feedback and/or Correction Requests and
    2. Most importantly decides whether to Accept or Reject the task as a good final annotation in accordance with the guideline.
  3. An accepted task is done and soon ready to download by the client.
    1. In the future Annotators will also receive feedback from Accepted tasks, today that's not supported yet.
  4. A rejected annotation is assigned to an Annotator, if possible the original one, to:
    1. Correct the Correction Requests and
    2. Acknowledge or act on the Feedback
    3. Reply to both of the above, if needed
    4. Submits the corrected annotation
  5. A reviewer reviews again and she still Rejects step 4 is repeated until she Accepts.

Correction request vs feedback

In our review phase we use two types of feedback: Correction requests and Advice.

Correction requests

A correction request represents something the person reviewing a task want to have corrected. This means if an annotator recieves a correction task with correction requests in it - she is expected to correct the error the reviewer have indicated.

Advice

These items represents something the person reviewing wants the annotator to know or think about, but not something they expect you to correct. Advice could be things such as encouraging words and general tips on how to improve.

ο»Ώ

Feedback Conversations

Discussing on the right way to do annotations can quickly become complex and details start to matter. Your feedback conversations might be much better if you consider the following tipps.

Giving feedback

  • Concentrate on the annotation content, not the person
    • For example, "you forgot to add a box here" vs. "I think that according to the guidelines there should be a box for this parked car"
  • Also give positive, encouraging feedback
  • Try to be specific
    • For example "behind the trees there's some parking cars missing" vs. "To me it seems there's five parking cars behind the trees"
  • Feedback is your personal opinion and might be debateable, phrase it as such

Receiving feedback

  • Be aware of responses
  • Be open to different opinions
  • Ask for clarifications if you don't understand how to act on feedback

ο»Ώ