Client QA

the client qa phase is part of the kognic standard workflow docid\ seukel bw5wxsob2z2tuk , which aims to enable the delivery of sufficient quality annotations on time and at the expected cost overview the client qa phase is a structured handoff stage that sits between the production flow and final delivery it gives clients a formal opportunity to review annotation quality, provide feedback, and submit a verdict before delivery is confirmed if quality does not meet the agreed acceptance criteria, a structured alignment and rework process ensures issues are resolved efficiently before the next review the client qa phase fits into the kognic standard workflow as follows annotate → quality review → expert verification → client qa → delivery when to use this phase client qa is used when a client wants to verify that delivered annotations meet agreed quality standards before the data is considered final workflow variants when you create a request, you choose which client qa workflow to use client qa the standard workflow after quality is accepted, corrected inputs move directly to delivery client qa with follow up review after quality is accepted, rejected inputs go through a re review loop before reaching delivery this gives the client a chance to verify that corrections meet their expectations, even when they accepted the full delivery both workflows share the same review, verdict, and rework stages described on this page the only difference is what happens to rejected inputs after quality is accepted see follow up review under rework accepted quality for details end state client qa ends when the overall quality verdict is accepted, either directly by the client, or jointly after verdict verification once accepted, remaining corrections are completed and all inputs move to the delivery phase if quality is rejected, additional client qa phases may follow rework until acceptance is reached phase stages a single client qa phase consists of the following stages client review the client samples and reviews annotations, then submits a quality verdict verdict verification (only when the client rejects) the producer reviews the rejection rationale and either verifies or adjusts the verdict rework the producer corrects annotations based on the joint verdict delivery accepted and corrected inputs are delivered ℹ️ note that the flow chart shows the regular client qa phase if you want to see how the flow changes when follow up review is activated, you can do so in the section rework accepted quality stage client review what happens here the client team reviews a representative sample of the produced annotations to determine whether the overall annotation quality meets the communicated acceptance criteria how to sample inputs inputs can be sampled in two ways manual sampling select individual inputs or a batch of inputs from the phase inputs table the table shows supporting metrics, including the number of objects and shapes per input automatic input sampling configure an automatic sampling level the platform tracks both actual sample size (inputs where review is complete) and planned sample size (inputs completed, in review, or waiting for review) across inputs, objects, and shapes quality metrics during client review, feedback based quality metrics are calculated for the reviewed annotations and made visible to both the client and the producer the metrics are provided both on an aggregated phase level and individually for all inputs that have been reviewed the following metrics are available metric aggregated (phase level) per input reviewed shapes total number of annotated shapes in inputs that have completed review includes false positives, excludes false negatives shapes with feedback percentage of reviewed shapes with feedback attached (shapes with feedback / reviewed shapes) × 100 percentage of reviewed shapes with feedback attached false positives percentage of reviewed shapes given feedback with the error type "not an object" (shapes with "not an object" feedback / reviewed shapes) × 100 percentage of reviewed shapes given feedback with the error type "not an object" false negatives percentage of missed shapes during annotation (error type "missing object") ("missing object" feedback items / (reviewed shapes + "missing object" feedback items)) × 100 percentage of missed shapes during annotation (error type "missing object") shapes with geometry errors percentage of reviewed shapes with feedback for "incorrect dimension", "incorrect position", or "incorrect layer order" (shapes with these error types / reviewed shapes) × 100 percentage of reviewed shapes with feedback for "incorrect dimension", "incorrect position", or "incorrect layer order" shapes with property errors percentage of reviewed shapes with feedback for "incorrect class" or "properties" (shapes with these error types / reviewed shapes) × 100 percentage of reviewed shapes with feedback for "incorrect class" or "properties" scene property errors the percentage of scene property errors across all frames and sources the percentage of scene property errors across all frames and sources scene properties describe the scene (e g , weather, vehicle speed, lighting) each property type contributes potential errors based on its scope static global 1 potential error per property (applies to entire scene) static source specific 1 potential error per property per source dynamic global 1 potential error per property per frame dynamic source specific 1 potential error per property per frame per source error rate = (total actual errors) / (total potential errors) × 100 the percentage of scene property errors across all of the input’s frames and sources quality metrics are calculated from feedback items for metrics to be meaningful, the client must provide structured, categorized feedback during review feedback without proper categorization will not be reflected in the metrics see how to write feedback for trustworthy quality metrics docid 9vbd9s1dxegzryyowpqdo for best practices disputed feedback items are excluded from all quality metric calculations submitting a verdict once the client has reviewed a sufficiently sized sample and evaluated the annotation quality, they submit a verdict accept the overall quality meets the acceptance criteria only the specific errors found during review need to be corrected reject the overall quality does not meet acceptance criteria a broader rework is required when a verdict is submitted, all open and ongoing review tasks are closed, any in progress work on those tasks is lost, and no new inputs can be selected for review the phase then moves in one of two directions reject → verdict verification accept → rework (accepted quality) stage verdict verification this stage only occurs when the client submits a reject verdict what happens here the producer reviews the client's rejection rationale, the feedback based quality metrics, and individual feedback items to determine whether the rejection is well founded the producer may also contact the client to clarify any feedback or resolve misunderstandings based on this review, the producer reaches a joint quality verdict verify (rejected) the producer agrees that the quality does not meet acceptance criteria the phase moves to rework rejected quality adjust (accepted) following review and clarification with the client, the producer determines the quality meets acceptance criteria the joint verdict becomes accepted, and the phase moves to rework accepted quality disputing feedback items before submitting the joint verdict, the producer can dispute individual feedback items they believe are incorrect only users with the producer role can initiate disputes disputes are managed through the correction request table in the client qa tab, where one or more items can be disputed at a time disputed feedback is excluded from quality metrics and correction tasks — which may change the overall quality picture and inform the joint verdict either the producer or the client can dismiss a dispute before the joint verdict is submitted, which reinstates the feedback in metrics and corrections both disputing and dismissing require a written reason once the joint verdict is submitted, any remaining disputes are final — disputed items are permanently excluded from metrics and corrections disputing feedback items is a phase level action in client qa — it is separate from task level feedback interactions such as replying to, resolving, or marking feedback as invalid in the task view stage rework the rework stage differs depending on whether the joint verdict is rejected or accepted rework rejected quality when the joint verdict is rejected, no inputs are sent for correction automatically you select what needs correcting in the phase inputs table a rejected quality verdict does not reject the individual inputs inside it each input keeps its own review verdict, so a phase whose quality was rejected still contains inputs the client accepted selecting inputs for correction selecting an input sends it for correction every input in the phase has one of two outcomes selected a correction task is created the input moves to the next client qa phase as soon as that correction is submitted not selected the input is untouched it moves to the next client qa phase when you mark the selection as completed you must select every input the client rejected you cannot mark the selection as completed until they have all been sent for correction inputs the client accepted or did not review do not have to be corrected, and are usually left unselected use the client's rejection rationale and the quality metrics to decide whether any of them need work as well, which is more common for non reviewed inputs when sending inputs for correction, you can attach additional feedback this feedback is applied to all currently selected inputs and appears as a feedback item in the resulting correction tasks use this to communicate high level instructions, such as systematic issues identified during client review input information available review status (rejected, accepted, or not reviewed by the client) number of objects and shapes completing the selection mark selection as completed is a phase level action there is no way to mark an individual input as done the one action moves every input that is not in correction to the next client qa phase at once you will find it in two places during rework step 3 of the rework to do list, and the top right of the rework selection panel above the phase inputs table finish selecting before you use it anything still unselected at that point moves to the next phase uncorrected inputs that are not selected do not advance on their own they stay in the phase until you mark the selection as completed marking the selection as completed does not affect inputs already in correction those move on as soon as their correction tasks are submitted, whether or not you have marked the selection as completed after rework, a new client qa phase is created, allowing the client to review the corrected annotations the new phase starts a fresh client review, so the client reviews from scratch rework accepted quality when the joint verdict is accepted (either directly from client review, or revised to accepted during verdict verification), inputs are handled as follows this describes the standard client qa workflow, see follow up review below for that variant client rejected inputs automatically sent for correction client accepted or non reviewed inputs move directly to the delivery phase and are considered delivered immediately once correction tasks are completed, the corrected inputs also move to the delivery phase once all inputs reach delivery, the request is complete follow up review when you select the client qa with follow up review workflow for a request, rejected inputs don't move straight to delivery after correction instead, a follow up phase is added between client qa and delivery in the follow up phase, the client re reviews corrected inputs inputs cycle through review → correct until the client accepts them accepted and non reviewed inputs are not affected, they move to delivery as usual delivery once all inputs have been accepted and any required corrections have been completed, they move to the delivery docid\ r2tntqu snfl4yggzk1dj phase the request is then considered complete frequently asked questions when does a new client qa phase start? a new client qa phase is created when the joint verdict in the current phase is rejected the next phase allows the client to review the corrected annotations what extra information is available when sampling in a second or later client qa round? from round 2 onward, the phase inputs table shows two additional pieces of information per input whether it was accepted, rejected, or not reviewed in a previous round, and whether it was corrected this helps the client focus their sampling on the most relevant inputs, for example, prioritising those corrected since the last review what happens to inputs the client did not review? non reviewed inputs are treated the same as accepted inputs when the overall verdict is accepted they move directly to the delivery phase why does the quality metrics look incomplete or lower than expected? quality metrics rely on feedback items being properly categorized make sure each feedback item has a defined error type unstructured comments without categorization will not be included in the metric calculations can the client add more inputs to review after submitting a verdict? no once a verdict is submitted, all open review tasks are closed and no additional inputs can be selected for review in that phase what happens when a feedback item is disputed? disputed feedback items are excluded from quality metrics and correction tasks only users with the producer role can initiate a dispute, and only through the correction request table in the client qa tab — not from the task view before the joint verdict is submitted, either the producer or the client can dismiss a dispute, which reinstates the feedback both disputing and dismissing require a written reason once the joint verdict is submitted, any remaining disputes are final — the excluded items are permanently removed from metrics and will not generate correction tasks who can dispute feedback items in client qa? only the producer can initiate disputes, and only through the correction request table in the client qa tab during the verdict verification stage this is not the same as replying to, resolving, or marking feedback as invalid in the task view — those are task level actions available to annotators and reviewers disputing is a phase level action that excludes feedback from quality metrics and correction tasks what is follow up review? follow up review is a client qa workflow variant you select when creating a request when this workflow is active, rejected inputs go through a re review loop after correction, and the client reviews them again before they reach delivery does follow up review affect accepted or non reviewed inputs? no only rejected inputs enter the follow up phase accepted and non reviewed inputs move to delivery as usual, regardless of which workflow you selected the quality verdict was rejected do the inputs the client accepted move on automatically? no when the joint verdict is rejected, nothing moves on automatically accepted and non reviewed inputs stay in the phase until you mark the selection as completed they move on by themselves only when the verdict is accepted related pages kognic standard workflow docid\ seukel bw5wxsob2z2tuk how to write feedback for trustworthy quality metrics docid 9vbd9s1dxegzryyowpqdo the delivery phase docid\ r2tntqu snfl4yggzk1dj