A review phase acts as a quality control step inside the request workflow, where all annotated inputs or a selection of them need to be approved by a reviewer to leave the phase. Rejected annotations are sent for correction and then re-reviewed.

There are multiple different types of review phases available: Full review, Sampled review, Quality Review, Expert Verification, and Client QA.

In phases of the type "Sampled review" only a number of the annotated inputs (intermediate annotations) that enter the phase are initially selected for review (approximately 20 % in the flowchart above). The inputs not selected for review stay in the phase until a manager decides, based on the result of the sampled review, if the non-selected inputs should be reviewed or if they can be sent directly to the next phase without being reviewed and potentially corrected.

Annotated inputs selected for review (both initially selected and selected due to phase decision) must have their quality approved by a reviewer to complete and leave the phase. If the annotation quality isn't acceptable, a task with the action "correct" is created, and a labeler is asked to improve the quality according to the reviewer's feedback. The updated annotation is then reviewed again; if the quality isn't acceptable, the cycle is repeated until it is.

When entering the phase, each annotated input has a set likelihood of being selected for review. That means if the sample size is set to 20 %, each annotated input entering the phase has a 20 % likelihood of being selected for review. Thus, the final sample size could be just below or just above the configured sample size.

This phase contains a mandatory decision in which you, as a manager, based on the result of the initial review, determine what should happen to the annotated inputs that haven't been selected for initial review (incl. inputs that haven't entered the phase yet as well). You can either decide to send them directly to the next phase or send them all for review. You can read more about the decision in the section Decision below.

A sampled review phase can be configured as a Client sampled review, where the verdict comes from the client instead of an internal reviewer. For richer client review (quality metrics, formal verdicts, dispute handling), see Client QA in the Kognic Standard Workflow.

This sampled review phase contains a mandatory decision in which you, as a manager, determine what should happen to the annotated inputs that haven't been selected for initial review (including inputs that haven't entered the phase yet) based on the result of the initial review.

Making a decision

With these actions, you can ensure that specific phase inputs either move to the next phase without review or go directly to review correction.

Accepts and moves inputs with unstarted review tasks to the next phase without needing to be accepted by a reviewer inside a review task. Inputs with unstarted review tasks have the Task type "Review" and current task state "To Do".

Rejects inputs with unstarted review tasks without requiring a reviewer to reject it inside a review task. When rejecting a review correction task is created, which is automatically assigned to the original annotator. Inputs with unstarted review tasks have the Task type "Review" and current task state "To Do".

If you want to view or edit the task allocation of review and correct tasks, you can do that inside the tab Tasks.

Det ser bra ut. Lägg in det så justerar jag resten själv.

Use Send to next phase to move inputs that are waiting for a decision directly to the next phase without creating a review task. The annotation moves forward unchanged.

Inputs waiting for a decision are inputs that have not been selected for review and do not have a task in the current phase. Normally, these inputs only leave the phase through the phase decision. With this action, you can move individual inputs forward without making a decision for the entire phase.

You can use the search bar or the input ID filter to find specific inputs.

In this section, you can monitor the current progress of the review phase and answer questions such as:

The number of inputs is currently in the phase.

The percentage out of all inputs that have completed and left the phase. In review phases, "completed" means that the input has been accepted by a reviewer, either via a review task or a quick accept action.

In this graph, you get insight into how the requests inputs are distributed in relation to this phase and its workflow stages. You can see how many inputs are in an earlier phase, how many are inside the phase, and in which workflow stage they are, as well as how many have been accepted and left the phase.

In the review phase, an annotated input can pass through multiple review rounds, depending on how many reviews and corrections it takes for it to become acceptable quality. Each round contains a cycle of correction and review, with the exception of round 1, which only contains a review step. When the annotated input is rejected, it moves to a new round.

In the card Distribution of inputs in rounds, you can view how the different inputs are currently spread throughout different states and rounds. Per round, you can see how many annotated inputs are currently in correction, in review (first review or re-review), or have been accepted and are, therefore, delivery-ready.

The chart visualizes the absolute number and percentage of accepted and rejected review tasks per round, which helps you understand the acceptance ratio of annotations entering the phase as well as in later rounds.

In the phase input table, you can see all inputs that are currently inside the phase. For each input, you can see when it changed workflow stage, how many tasks have been done on it in the current phase, and what the state and type is of any current task.

The actions Quick Accept and Quick Reject are available for inputs with unstarted review tasks. You can read more about them in the section Actions above.

The action Send to next phase is available for inputs that are waiting for a decision. You can read more about it in the section Actions above.

With the Error summary you get insight into what issues reviewers have found and commented on during the phase's review tasks. It helps you understand the most common and less frequent identified issues.

The number of feedback items of the type "Correction Requests". This is the sum of all errors shown in the "Error Type Distribution" to the right.

The number of feedback items of the type "Advice". These are excluded from the chart "Error Type Distribution" and "Suggested properties".

Shows the absolute count and relative share of all feedback items categorized as "Correction Requests", grouped by their error type.

For those items with the error type Properties, this shows the distribution of properties that we affected. Each error indicated as Properties has a single property connected to it.

This section helps you to get an overview of all given feedback, to answer questions such as:

The items are split up by their feedback type.

In this section, you see feedback items for the type Correction Request. These are things that the reviewer wants to get corrected before accepting the review.

You can filter the feedback items by their Resolved status, the Error type, whether a discussion thread exists, or whether the overall Review of the input has been accepted yet.

Below is a description of what information is available for each correction request.

Status

An item can be "Unresolved" even if the overall Review was accepted, or the other way around.

Error Type The type of error that was selected in the Correction Request.

Suggested property If the Error type is "Properties", this column shows which property and value was suggested by the reviewer.

Comment Shows the description that the reviewer might have given.

Thread exists Will say "Yes", if there was any reply to the item, i.e. a discussion thread has been started in relation to the item.

External Scene ID The scene ID of the reviewed annotation.

Current Round The review round in which the input of this feedback item currently is. All inputs start in round 1. With each rejected review, they progress 1 round forward.

Accepted Review Whether the overall Review was accepted or not.

In this section you see feedback items of the type Advice. As the underlying data has less structure, the table has fewer columns and filtering options, but otherwise, it looks the same as the one above.