Client QA

the client qa phase is part of the docid\ seukel bw5wxsob2z2tuk , which aims to enable the delivery of sufficient quality annotations on time and at the expected cost overview the client qa phase is a structured handoff stage that sits between the production flow and final delivery it gives clients a formal opportunity to review annotation quality, provide feedback, and submit a verdict before delivery is confirmed if quality does not meet the agreed acceptance criteria, a structured alignment and rework process ensures issues are resolved efficiently before the next review the client qa phase fits into the kognic standard workflow as follows annotate → quality review → expert verification → client qa → delivery when to use this phase client qa is used when a client wants to verify that delivered annotations meet agreed quality standards before the data is considered final end state client qa ends when the overall quality verdict is accepted, either directly by the client, or jointly after verdict verification once accepted, remaining corrections are completed and all inputs move to the delivery phase if quality is rejected, additional client qa phases may follow rework until acceptance is reached phase stages a single client qa phase consists of the following stages client review the client samples and reviews annotations, then submits a quality verdict verdict verification (only when the client rejects) the producer reviews the rejection rationale and either verifies or adjusts the verdict rework the producer corrects annotations based on the joint verdict delivery accepted and corrected inputs are delivered stage client review what happens here the client team reviews a representative sample of the produced annotations to determine whether the overall annotation quality meets the communicated acceptance criteria how to sample inputs inputs can be sampled in two ways manual sampling select individual inputs or a batch of inputs from the phase inputs table the table shows supporting metrics, including the number of objects and shapes per input automatic input sampling configure an automatic sampling level the platform tracks both actual sample size (inputs where review is complete) and planned sample size (inputs completed, in review, or waiting for review) across inputs, objects, and shapes quality metrics during client review, feedback based quality metrics are calculated for the reviewed annotations and made visible to both the client and the producer the metrics are provided both on an aggregated phase level and individually for all inputs that have been reviewed the following metrics are available true 220,220,221 unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type 1 1 unhandled content type important quality metrics are calculated from feedback items for metrics to be meaningful, the client must provide structured, categorized feedback during review feedback without proper categorization will not be reflected in the metrics see docid 9vbd9s1dxegzryyowpqdo for best practices submitting a verdict once the client has reviewed a sufficiently sized sample and evaluated the annotation quality, they submit a verdict accept the overall quality meets the acceptance criteria only the specific errors found during review need to be corrected reject the overall quality does not meet acceptance criteria a broader rework is required when a verdict is submitted, all open review tasks are closed, and no new inputs can be selected for review the phase then moves in one of two directions reject → verdict verification accept → rework (accepted quality) stage verdict verification this stage only occurs when the client submits a reject verdict what happens here the producer reviews the client's rejection rationale, the feedback based quality metrics, and individual feedback items to determine whether the rejection is well founded the producer may also contact the client to clarify any feedback or resolve misunderstandings based on this review, the producer reaches a joint quality verdict verify (rejected) the producer agrees that the quality does not meet acceptance criteria the phase moves to rework rejected quality adjust (accepted) following review and clarification with the client, the producer determines the quality meets acceptance criteria the joint verdict becomes accepted, and the phase moves to rework accepted quality stage rework the rework stage differs depending on whether the joint verdict is rejected or accepted rework rejected quality when the joint verdict is rejected , no inputs are automatically sent for correction instead, the producer manually selects which inputs to correct based on the client's rejection rationale selecting inputs for correction selection is manual via the phase inputs table one or more inputs can be sent for correction at a time when sending inputs for correction, you can attach additional feedback this feedback is applied to all currently selected inputs and appears as a feedback item in the resulting correction tasks use this to communicate high level instructions, such as systematic issues identified during client review input information available review status (rejected, accepted, or not reviewed by the client) number of objects and shapes completing the selection once you have finished selecting all inputs for correction, you confirm this by marking the selection as complete this ensures that inputs that shouldn't be corrected move to the next client qa phase selected inputs proceed once their correction tasks are completed; marking the selection as complete does not affect them, only the non selected inputs after rework, a new client qa phase is created, allowing the client to review the corrected annotations the new phase starts a fresh client review, so the client reviews from scratch rework accepted quality when the joint verdict is accepted (either directly from client review or via an adjusted verdict in verdict verification), inputs are handled as follows client rejected inputs automatically sent for correction client accepted or non reviewed inputs move directly to the delivery phase and are considered delivered immediately once correction tasks are completed, the corrected inputs also move to the delivery phase once all inputs reach delivery, the request is complete delivery once all inputs have been accepted and any required corrections have been completed, they move to the docid\ r2tntqu snfl4yggzk1dj phase the request is then considered complete frequently asked questions when does a new client qa phase start? a new client qa phase is created when the joint verdict in the current phase is rejected the next phase allows the client to review the corrected annotations what extra information is available when sampling in a second or later client qa round? from round 2 onward, the phase inputs table shows two additional pieces of information per input whether it was accepted, rejected, or not reviewed in a previous round, and whether it was corrected this helps the client focus their sampling on the most relevant inputs, for example, prioritising those corrected since the last review what happens to inputs the client did not review? non reviewed inputs are treated the same as accepted inputs when the overall verdict is accepted they move directly to the delivery phase why does the quality metrics look incomplete or lower than expected? quality metrics rely on feedback items being properly categorized make sure each feedback item has a defined error type unstructured comments without categorization will not be included in the metric calculations can the client add more inputs to review after submitting a verdict? no once a verdict is submitted, all open review tasks are closed and no additional inputs can be selected for review in that phase related pages docid\ seukel bw5wxsob2z2tuk docid 9vbd9s1dxegzryyowpqdo docid\ r2tntqu snfl4yggzk1dj