Quality Alignment

background quality alignment indicates how well annotators work aligns with expected quality standards it measures the accuracy of annotation work by comparing it against review corrections and ground truth data this page helps quality managers and team leads monitor individual annotator performance across key quality metrics identify training opportunities by spotting consistent patterns in errors (missing shapes, incorrect classifications, geometry inaccuracies, property mistakes) track quality trends over time to see if performance is improving or decreasing make data driven decisions about team assignments, training needs and quality processes how it works quality alignment metrics are calculated by analyzing the edits made during the review phase when an annotator's work is corrected , these corrections are tracked across four key dimensions precision are annotators labeling shapes correctly, or are they frequently mislabeling things? formula (correct identifications) / (total identifications made) recall are annotators finding all the shapes they should, or are they missing important items? formula (correct identifications) / (total shapes that should have been found) geometry accuracy are shapes, sizes, positions, and orientations correct? formula 1 (edited shapes / reviewed shapes) property accuracy are shape attributes and properties assigned correctly? formula 1 (edited properties / reviewed properties) the alignment score provides a single overall quality indicator by taking the minimum of these four metrics this ensures that weaknesses in any dimension are reflected in the overall score the quality score provides an overall quality indicator by averaging these four metrics unlike the alignment score which takes the minimum value, the quality score gives a balanced view of overall performance across all dimensions shape based calculations we are using a shape based method instead of an object based method to evaluate each individual geometric shape independently, providing granular accuracy measurements benefits higher precision captures partial errors within multi shape objects fairer evaluation accuracy reflects actual geometric work, not just object level decisions better training data identifies specific geometric issues rather than broad object level failures when to use this use quality alignment when you need to evaluate annotator readiness for production work investigate quality issues reported by reviewers or clients compare performance across team members track the effectiveness of training programs identify which types of errors (class confusion, geometry issues, property mistakes) are most common components alignment score trend a trend graph showing the selected quality metric over time with currently applied filters, displaying the average value per day can be further filtered on specific users hovering over data points displays the metric value with a % suffix and the number of samples use the metric selector in the graph header to switch between quality score (default) — average of precision, recall, geometry accuracy, and property accuracy alignment score — minimum of the four sub metrics precision — ratio of correct identifications to total identifications recall — ratio of correct identifications to total expected shapes geometry accuracy — shape and boundary accuracy property accuracy — attribute assignment accuracy user quality alignment table assignment quality alignment table