How to write feedback for trustworthy quality metrics
During Client Review, your feedback is not just a communication tool; it is the foundation for the quality metrics used to evaluate the overall annotation quality and inform the accept/reject verdict. This guide explains how to write feedback that produces accurate, meaningful metrics.
Why feedback quality matters
Quality metrics in Client QA are calculated directly from the feedback items you submit during review. If feedback is missing, incomplete, or not properly categorized, the metrics will be incomplete or misleading β and may not reflect the true quality of the delivery.
A verdict based on inaccurate metrics makes it harder to reach a shared understanding of quality between you and the producer, and can lead to unnecessary rework or missed issues.
How to add feedback
There are three ways to add a feedback item in the task view. For efficiency, Quick feedback is recommended. It lets you tag errors without having to fill out the full feedback form.
Quick feedback (recommended)ο»Ώ
ο»ΏQuick feedbackο»Ώ is the fastest way to log errors during review.
Activate Quick feedback with Ctrl+X, or click the Quick feedback icon in the expandable sidebar.
Click directly on an object or a location in the scene. The tool auto-detects whether you're flagging an object or placing a location pin.
A feedback chip appears next to your cursor. Use keys 1β9 to rapidly select the error type, then optionally add a comment.
You can hold Ctrl/Cmd to multi-select objects and leave feedback on several at once.
Object feedback
Use Object feedbackο»Ώ when you need more control, for example to explicitly set the time scope (current frame or entire sequence) and sensor scope (current sensor or all sensors).
- With an object selected: Open the feedback form, and the object is automatically linked.
- Without a selection: Open the feedback form, click Select objects, then click one or more objects in the main view or edit tab.
Location feedback (pins)
Use Location feedback (pins)ο»Ώ to mark a position in the scene without referencing an existing object. This is the right approach for missing objects, see the dedicated section below.
Open the feedback form, click Place pin, then click the location of the issue in the 2D or 3D view. The current frame and sensor are auto-filled, and you can clarify the scope further in the comment field.
How to apply feedback: time and sensor scope
When using Object feedback, you explicitly control the time and sensor scope of a feedback item. This directly affects how errors are counted in the quality metrics.
Time scope:
- This frame -> the error applies to the object in the current frame only.
- All frames -> the error applies to the object across the entire sequence.
Sensor scope:
- This sensor -> the error applies to the object in the current sensor only.
- All sensors -> the error applies to the object across all sensors.
Together, these choices define how many shapes the feedback item covers:
Time | Sensor | Errors counted |
|---|---|---|
This frame | This sensor | 1 (a specific shape) |
This frame | All sensors | One per sensor in that frame |
All frames | This sensor | One per frame for that sensor |
All frames | All sensors | One per frame Γ sensor |
Key principle: Scope feedback as precisely as the error warrants. A misplaced box in one camera in one frame should be scoped to that frame and sensor. A wrong class label that applies across the entire sequence should be scoped to all frames and all sensors; the metrics will count it across all shapes accordingly.
Error types and what they measure
Always select the correct error type when submitting a correction request. Each error type maps to a specific quality metric. Using the wrong type means the error will either not be counted (Other or Advice) or will be attributed to the wrong metric.
Error type | What it means | Metric it affects |
|---|---|---|
Not an object | The annotated shape should not exist; it is a false detection | False positives |
Missing object | An object that should have been annotated was missed | False negatives |
Incorrect dimension | The shape size is wrong (e.g. bounding box is too large or too small) | Shapes with geometry errors |
Incorrect position | The shape is placed at the wrong location | Shapes with geometry errors |
Incorrect layer order | The depth ordering of overlapping shapes is wrong | Shapes with geometry errors |
Incorrect class | The object is annotated with the wrong class label | Shapes with property errors |
Properties | One or more attribute values on the object are wrong | Shapes with property errors |
Properties (on a scene property) | A scene-level property value is wrong | Scene property errors |
For Incorrect class and Properties, the UI will prompt you to suggest the correct class or property value. Providing this makes the feedback much more actionable for the producer during rework.
How to report a missing object
Reporting a missing object is different from other feedback types because there is no existing annotation to click on since the shape is absent by definition. Use location feedback (pins) for this.
Navigate to the frame where the object should have been annotated.
Activate Quick feedback (Ctrl+X), then click the location in the scene where the object should be, or open the feedback form manually and click Place pin.
Set the error type to Missing object.
Add a comment describing what is missing and where, for example: "Pedestrian on the left side of the road, approximately 15 m from ego vehicle, not annotated."
Missing-object feedback is used to calculate the false-negative rate: the proportion of objects that should have been annotated but were not. The more precisely you describe the location and nature of the missing object, the more actionable the feedback is for the producer.
Common mistakes to avoid
Writing feedback of the type "Advice" instead of "Correction request". Advice allows you to write encouraging words and general tips for improvement. An advice item should not tell the annotator to fix anything in the current task. Therefore, feedback of the type 'advice' isn't counted as an error in the feedback-based quality metrics in Client QA. Instead, always select "Correction request" when you want to point out an actual error.
Using the error type "Other". This error type lets you flag an error with only a written comment. Because correction requests of this type are free-text only, they aren't counted in the feedback-based quality metrics.
Over-scoping feedback. If a box is misplaced in one specific frame and camera, scope the feedback to that frame and sensor only. Setting all frames and all sensors will inflate the error count across the entire sequence.
Under-scoping feedback. If the same error occurs across all frames, scope the feedback to all frames, not just the current frame.
Related pages
ο»Ώ
ο»ΏClient QAο»Ώ
ο»ΏAdd feedbackο»Ώ
ο»ΏQuick-Feedbackο»Ώ
ο»ΏObject feedbackο»Ώ
ο»ΏLocation feedback (pins)ο»Ώ
ο»ΏError Typesο»Ώ