In the Kognic platform, annotations can be produced by using multiple different workflows. These workflows determine the steps used to produce the annotations and in what order. They define what type of steps (tasks) to complete before we have produced a deliverable annotation.

Which workflow suits each request is determined by multiple factors, such as the desired quality, time, and cost.

A workflow consists of an annotation phase and, optionally, a review phase.



When creating a new request you have to select which workflow to use. The workflows available for selection both use the annotation phase AC. The annotation phase CQC can currently only be added to the workflow by Kognic.

You can read more about creating requests and selecting their workflow in the guides: Guide: Create a request and Guide: Create a new project.

A task is what ties an input to a team member and what type of action the team member should take with the input. There are three types of tasks - annotate, correct, and review. On its way from original input to a delivery-ready (or final) annotation, the input spawns one or multiple tasks, sometimes even multiple tasks of the same type. The number of tasks depends on the workflows used in the request and if tasks expire and get assigned to new users.

Annotate An annotate task will be given to a team member with the purpose of them annotating it from scratch. It is always the initial task inside the annotation workflow. Correct A task where the team member is asked to correct a task with existing annotations. Depending on the workflow, a Correct task follows either each Annotate task, randomly sampled Annotate tasks, or such that got rejected during a Review/QA task.

In the Kognic platform, we have multiple different types of correction tasks, you can read more about them under Tasks. Review Review tasks are similar to Correct tasks but allow a team member to give structured feedback on objects, specific timestamps, and specific sensors inside the app. Review tasks also include a final judgement on whether to Accept or Reject the reviewed annotation

For a user to be able to work on a task it has to be assigned to the user. The user can now complete the assignment or let it expire. Once it expires, the task becomes available for assignment to other users.

The task is assigned either automatically by our system or manually by a manager, see Team for more details.

An annotation workflow consists of one annotation phase and an optional review phase. Below is a description of our two annotation phases.

The AC phase is straightforward. In the basic version, it just requires one annotate step per input. Optionally you can require one correction round per input, so each annotation gets checked and corrected once by another annotator. After the annotation and correction are completed, the annotated inputs move into the review phase (if part of the workflow) or become delivery ready.

The AC phase is used in the workflows we offer when creating a request, Annotate + Review, and Annotate + Correct + Review.



The CQC phase enables an approximation of a request's quality using sampled Quality Assurance (QA) tasks by dedicated Quality Experts. These QA tasks result in Quality KPIs that are used to approximate and reason about the quality of all annotations.



The CQC phase works as follows:

CQC also allows splitting the input batch into smaller Annotation batches. Each annotation batch goes through the CQC loop above independently, allowing the manager to deliver smaller increments more often.

Quality Assure tasks are correction tasks where the quality manager/expert is asked to ensure the annotation is of the highest possible quality. This creates a diff between the original and the corrected annotation, which is then used to calculate quality KPIs in a flexible and automated way.

These corrections are an alternative to using a second from-scratch annotation (blank reference annotation) for the comparison. By using corrections, we can reduce the number of mismatches purely based on ambiguity rather than actual annotation errors as defined by the guideline.

The review phase is an optional add-on to an annotation phase. When enabled, it intercepts each annotated input that would usually become a Delivery Ready annotation and sends it for review by a Reviewer. Each annotated input needs to be accepted during the Review to make it delivery ready and ready for download.

An essential characteristic of the Review Phase is that it allows for giving structured feedback on the annotation. You can read more about this in the Feedback Tools chapter.

The review phase The review phase, if enabled, starts once an intermediate annotation exists (i.e., an annotated input that has passed through the steps of the annotation phase). When the intermediate annotation is available, the flow continues as follows:



Some settings are valid across all workflows, primarily those configuring a specific task's flow.

The duration between task assignment or last update in the annotation (whatever is later), after which the task is expired and is no longer available to the assigned user. When the task is expired, a new task will be created with the same initial states as the original one. This means that work done on the expired task won’t be transferred to the new tasks nor become part of the final annotation.