The Client QA Phase is part of the Kognic Standard Workflow, which aims to enable the delivery of sufficient-quality annotations on time and at the expected cost.

The Client QA phase is a structured handoff stage that sits between the production flow and final delivery. It gives clients a formal opportunity to review annotation quality, provide feedback, and submit a verdict before delivery is confirmed. If quality does not meet the agreed acceptance criteria, a structured alignment and rework process ensures issues are resolved efficiently before the next review.

The Client QA phase fits into the Kognic Standard Workflow as follows:

Client QA is used when a client wants to verify that delivered annotations meet agreed quality standards before the data is considered final.

When you create a request, you choose which Client QA workflow to use:

Both workflows share the same review, verdict, and rework stages described on this page. The only difference is what happens to rejected inputs after quality is accepted. See Follow-up Review under Rework: Accepted quality for details.

Client QA ends when the overall quality verdict is Accepted, either directly by the client, or jointly after Verdict Verification. Once accepted, remaining corrections are completed and all inputs move to the Delivery phase.

If quality is rejected, additional Client QA phases may follow rework until acceptance is reached.

A single Client QA phase consists of the following stages:

ℹ️ Note that the flow chart shows the regular Client QA phase. If you want to see how the flow changes when Follow-up review is activated, you can do so in the section Rework: Accepted quality.

The client team reviews a representative sample of the produced annotations to determine whether the overall annotation quality meets the communicated acceptance criteria.

Inputs can be sampled in two ways:

During Client Review, feedback-based quality metrics are calculated for the reviewed annotations and made visible to both the client and the producer. The metrics are provided both on an aggregated phase level and individually for all inputs that have been reviewed. The following metrics are available:

Once the client has reviewed a sufficiently sized sample and evaluated the annotation quality, they submit a verdict:

When a verdict is submitted, all open and ongoing review tasks are closed, any in-progress work on those tasks is lost, and no new inputs can be selected for review. The phase then moves in one of two directions:

The producer reviews the client's rejection rationale, the feedback-based quality metrics, and individual feedback items to determine whether the rejection is well-founded. The producer may also contact the client to clarify any feedback or resolve misunderstandings.

Based on this review, the producer reaches a joint quality verdict:

Disputing feedback items

Before submitting the joint verdict, the producer can dispute individual feedback items they believe are incorrect. Only users with the producer role can initiate disputes. Disputes are managed through the Correction Request table in the Client QA tab, where one or more items can be disputed at a time. Disputed feedback is excluded from quality metrics and correction tasks — which may change the overall quality picture and inform the joint verdict. Either the producer or the client can dismiss a dispute before the joint verdict is submitted, which reinstates the feedback in metrics and corrections. Both disputing and dismissing require a written reason. Once the joint verdict is submitted, any remaining disputes are final — disputed items are permanently excluded from metrics and corrections.

The Rework stage differs depending on whether the joint verdict is Rejected or Accepted.

When the joint verdict is Rejected, no inputs are sent for correction automatically. You select what needs correcting in the phase inputs table.

A rejected quality verdict does not reject the individual inputs inside it. Each input keeps its own review verdict, so a phase whose quality was rejected still contains inputs the client accepted.

Selecting an input sends it for correction. Every input in the phase has one of two outcomes:

Inputs the client accepted or did not review do not have to be corrected, and are usually left unselected. Use the client's rejection rationale and the quality metrics to decide whether any of them need work as well, which is more common for non-reviewed inputs.

When sending inputs for correction, you can attach additional feedback. This feedback is applied to all currently selected inputs and appears as a feedback item in the resulting correction tasks. Use this to communicate high-level instructions, such as systematic issues identified during Client Review.

Input information available

Mark selection as completed is a phase-level action. There is no way to mark an individual input as done. The one action moves every input that is not in correction to the next Client QA phase at once.

You will find it in two places during Rework: step 3 of the Rework to-do list, and the top right of the Rework selection panel above the phase inputs table.

Finish selecting before you use it. Anything still unselected at that point moves to the next phase uncorrected.

Inputs that are not selected do not advance on their own. They stay in the phase until you mark the selection as completed.

Marking the selection as completed does not affect inputs already in correction. Those move on as soon as their correction tasks are submitted, whether or not you have marked the selection as completed.

After rework, a new Client QA phase is created, allowing the client to review the corrected annotations. The new phase starts a fresh Client Review, so the client reviews from scratch.

When the joint verdict is Accepted (either directly from Client Review, or revised to Accepted during Verdict Verification), inputs are handled as follows. This describes the standard Client QA workflow, see Follow-up Review below for that variant.

Once correction tasks are completed, the corrected inputs also move to the Delivery phase. Once all inputs reach Delivery, the request is complete.

When you select the Client QA with Follow-up Review workflow for a request, rejected inputs don't move straight to Delivery after correction. Instead, a Follow-up phase is added between Client QA and Delivery.

In the Follow-up phase, the client re-reviews corrected inputs. Inputs cycle through Review → Correct until the client accepts them. Accepted and non-reviewed inputs are not affected, they move to Delivery as usual.

Once all inputs have been accepted and any required corrections have been completed, they move to the Delivery phase. The request is then considered complete.