How to write feedback for trustworthy quality metrics

during client review, your feedback is not just a communication tool; it is the foundation for the quality metrics used to evaluate the overall annotation quality and inform the accept/reject verdict this guide explains how to write feedback that produces accurate, meaningful metrics why feedback quality matters quality metrics in client qa are calculated directly from the feedback items you submit during review if feedback is missing, incomplete, or not properly categorized, the metrics will be incomplete or misleading — and may not reflect the true quality of the delivery a verdict based on inaccurate metrics makes it harder to reach a shared understanding of quality between you and the producer, and can lead to unnecessary rework or missed issues how to add feedback there are three ways to add a feedback item in the task view for efficiency, quick feedback is recommended it lets you tag errors without having to fill out the full feedback form quick feedback (recommended) (recommended) docid\ nrzwf0eqx6utnfyuvie8y is the fastest way to log errors during review activate quick feedback with ctrl+x , or click the quick feedback icon in the expandable sidebar click directly on an object or a location in the scene the tool auto detects whether you're flagging an object or placing a location pin a feedback chip appears next to your cursor use keys 1–9 to rapidly select the error type, then optionally add a comment you can hold ctrl/cmd to multi select objects and leave feedback on several at once object feedback use docid\ j6dvumpe0fwoqwqz qwfh when you need more control, for example to explicitly set the time scope (current frame or entire sequence) and sensor scope (current sensor or all sensors) with an object selected open the feedback form, and the object is automatically linked without a selection open the feedback form, click select objects , then click one or more objects in the main view or edit tab location feedback (pins) use docid\ ivbqxirb7mdty 7mbmxqr to mark a position in the scene without referencing an existing object this is the right approach for missing objects , see the dedicated section below open the feedback form, click place pin , then click the location of the issue in the 2d or 3d view the current frame and sensor are auto filled, and you can clarify the scope further in the comment field how to apply feedback time and sensor scope when using object feedback, you explicitly control the time and sensor scope of a feedback item this directly affects how errors are counted in the quality metrics time scope this frame > the error applies to the object in the current frame only all frames > the error applies to the object across the entire sequence sensor scope this sensor > the error applies to the object in the current sensor only all sensors > the error applies to the object across all sensors together, these choices define how many shapes the feedback item covers true 220,220,221left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type key principle scope feedback as precisely as the error warrants a misplaced box in one camera in one frame should be scoped to that frame and sensor a wrong class label that applies across the entire sequence should be scoped to all frames and all sensors; the metrics will count it across all shapes accordingly error types and what they measure always select the correct error type when submitting a correction request each error type maps to a specific quality metric using the wrong type means the error will either not be counted (other or advice) or will be attributed to the wrong metric true 220,220,221left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type for incorrect class and properties, the ui will prompt you to suggest the correct class or property value providing this makes the feedback much more actionable for the producer during rework how to report a missing object reporting a missing object is different from other feedback types because there is no existing annotation to click on since the shape is absent by definition use location feedback (pins) for this navigate to the frame where the object should have been annotated activate quick feedback (ctrl+x), then click the location in the scene where the object should be, or open the feedback form manually and click place pin set the error type to missing object add a comment describing what is missing and where, for example "pedestrian on the left side of the road, approximately 15 m from ego vehicle, not annotated " missing object feedback is used to calculate the false negative rate the proportion of objects that should have been annotated but were not the more precisely you describe the location and nature of the missing object, the more actionable the feedback is for the producer common mistakes to avoid writing feedback of the type "advice" instead of "correction request" advice allows you to write encouraging words and general tips for improvement an advice item should not tell the annotator to fix anything in the current task therefore, feedback of the type 'advice' isn't counted as an error in the feedback based quality metrics in client qa instead, always select "correction request" when you want to point out an actual error using the error type "other" this error type lets you flag an error with only a written comment because correction requests of this type are free text only, they aren't counted in the feedback based quality metrics over scoping feedback if a box is misplaced in one specific frame and camera, scope the feedback to that frame and sensor only setting all frames and all sensors will inflate the error count across the entire sequence under scoping feedback if the same error occurs across all frames, scope the feedback to all frames, not just the current frame related pages docid\ wrc okywzeq0kdyskb0mh docid\ dyap cmm72dmzpvz6e5lb docid\ nrzwf0eqx6utnfyuvie8y docid\ j6dvumpe0fwoqwqz qwfh docid\ ivbqxirb7mdty 7mbmxqr docid 7sa8 gk8s48crtht3ypmz