Every workflow ends in a delivery phase. When an input enters this phase, it becomes a delivery-ready annotation and completes the request. 🎉

In the Delivery phase, there are multiple ways to iterate further on an annotation, if needed. These are presented below.

Managers inside a requests producer organization of a request you can send a delivery-ready input for correction. This will move the input to the workflow stage Delivery: Correct and create a Correction task. The action will remove the Input from Delivery: Ready state but keep it in the same Workflow phase (Delivery).

If desired, you can assign the Correction task to a specific team member. However, this is an optional function and not selecting this means the task will be available for team members configured to work in the stage Delivery: Correct.

Managers of a request's producer or owner organizations can create review tasks for Delivery: Ready inputs.

Just after sending a delivery-ready input for review and while the input is in its first review round it will be part of both the workflow stages Delivery: Review and Delivery: Ready. This is because the delivery-ready annotation isn't replaced until it has been accepted or rejected in review.

If desired, you can assign the Review task to a specific team member. However, this is an optional function and not selecting this means the task will be available for team members configured to work in Delivery: Review.

Accepts and moves inputs with unstarted review tasks to the stage Delivery: Ready without needing to be accepted by a reviewer inside a review task. Inputs with unstarted review tasks have the Task type "Review" and current task state "To Do".

For Delivery: Ready inputs, you can download the Annotation in the OpenLabel format. You do this by opening the inputs option menu and clicking the button "Download annotation".

This section's content helps you understand how many inputs are delivery-ready and whether any additional iteration is being performed inside the delivery phase.

The number of inputs inside the delivery phase that currently are in the stages Delivery: Review and Delivery: Correct.

The percentage out of all inputs that are ready for delivery and download.

In this graph, you get insight into how the requests inputs are distributed in relation to this phase and its workflow stages. You can see how many inputs are in an earlier phase, how many are inside the phase, and in which workflow stage they are, as well as how many have are delivery-ready.

In the phase input table, you can see all inputs that are currently inside the delivery phase. For each input, you can see when it changed workflow stage, how many tasks have been done on it in the current phase, and the state and type of any current task.

The actions Send for review, Send for correction, and Quick Accept are available for specific inputs. You can read more about them in the section "Available actions" above.

The edit summary enables insight into how the annotations entering the delivery phase were adjusted as a consequence of reviews and corrections inside the delivery phase. These insights are gathered by comparing an input’s annotation when entering the delivery phase to when it once again becomes delivery-ready after being accepted in a review task. Note that quick-accepted inputs aren't used in this comparison, and any edits done in the phase for these inputs won't be included as edits in the edit summary.

Currently, you can investigate the edits in three different tables - Added and removed objects, Edited objects, and Edited scene properties.

Helps you understand how often specific objects were missed or incorrectly included, thus impacting the recall and precision of the annotations.

The table contains the following information per class:

Added objects The percentage of added objects seen in the review accepted inputs’ annotations compared to their content when entering the phase.

The numbers in this column enable insights such as:

💡 To ensure all relevant vehicles were annotated we had to add 20 % more objects during Delivery Review. We need to understand what objects were missed and how we can prevent that from happening in the future.

Removed objects The percentage of removed objects seen in the review accepted inputs’ annotations compared to their content when entering the phase.

The numbers in this column enable insights such as:

💡 We seem to have removed 50 % of all reviewed obstacle objects. Has the team misunderstood what defines an obstacle?

Objects after phase The percentage difference in object count between review-accepted inputs' annotations and their content at the start of the phase.

Object review ratio Compares the initial count of objects from review-accepted inputs (at phase entry) to the total number of objects from all inputs (incl. those currently in review correction and review) that have entered the phase. This helps you understand the current review sample of a specific class. ⚠️ Note that the review ratio is based on objects that have entered the phase; objects still in earlier phases are excluded. This means that the ratio doesn't represent the total object sample rate until all objects have completed the phase.

This table helps you understand how often properties and geometries had to be edited to meet the reviewers' quality expectations. Property edit and sample rates are presented per property, while geometry edits are presented per geometry type and type of edit (2D box - Position).

Edited objects The percentage of objects for which the attribute got edited between the annotated input entering the phase and being accepted in review.

💡 15 % of the reviewed 3D box objects have been resized; were the changes in size significant or just minor adjustments?

💡 35 % of the reviewed objects with the property "Age" had their property value changed. Is the definition of the different values unclear to the team?

Object review ratio Percentage of objects that are review-accepted compared to all objects in this phase. The value updates when inputs are reviewed and accepted, or when new inputs enter the phase.

⚠️ Note that the review ratio is based on objects that have entered the phase; objects still in earlier phases are excluded. This means that the ratio doesn't represent the total object attribute sample rate until all objects have completed the phase.

This table helps you understand how often individual scene properties had to be edited to meet the reviewers' quality expectations.

Edited inputs The percentage of inputs where the scene property was edited between initial phase entry and review acceptance.

💡 For 23 % of the reviewed inputs the scene property Weather got edited. Were there any particular property values that the team members had a hard time distinguishing in between?

Object review ratio The percentage of inputs that are review-accepted compared to all inputs in this phase. The value updates when inputs are reviewed and accepted or when new inputs enter the phase.

⚠️ Note that the review ratio is based on inputs that have entered the phase; inputs still in earlier phases are excluded. This means that the ratio doesn't represent the total input property sample rate until all inputs have completed the phase.

With the Error summary you get insight into what issues reviewers have found and commented on during the delivery phase's review tasks. It helps you understand the most common and less frequent identified issues.

The number of feedback items of the type "Correction Requests". This is the sum of all errors shown in the "Error Type Distribution" to the right.

The number of feedback items of the type "Advice". These are excluded from the chart "Error Type Distribution" and "Suggested properties".

Shows the absolute count and relative share of all feedback items categorized as "Correction Requests", grouped by their error type.

For those items with the error type Properties, this shows the distribution of properties that we affected. Each error indicated as Properties has a single property connected to it.

This section helps you to get an overview of all given feedback, to answer questions such as:

The items are split up by their feedback type.

In this section, you see feedback items for the type Correction Request. These are things that the reviewer wants to get corrected before accepting the review.

You can filter the feedback items by their Resolved status, the Error type, whether a discussion thread exists, or whether the overall Review of the input has been accepted yet.

Below is a description of what information is available for each correction request.

Status

An item can be "Unresolved" even if the overall Review was accepted, or the other way around.

Error Type The type of error that was selected in the Correction Request.

Suggested property If the Error type is "Properties", this column shows which property and value was suggested by the reviewer.

Comment Shows the description that the reviewer might have given.

Thread exists Will say "Yes", if there was any reply to the item, i.e. a discussion thread has been started in relation to the item.

External Scene ID The scene ID of the reviewed annotation.

Current Round The review round in which the input of this feedback item currently is. All inputs start in round 1. With each rejected review, they progress 1 round forward.

Accepted Review Whether the overall Review was accepted or not.

In this section you see feedback items of the type Advice. As the underlying data has less structure, the table has fewer columns and filtering options, but otherwise, it looks the same as the one above.