Annotation Requests
Under Project Management: Annotation Requests, you can find and navigate to non-archived annotation requests, by project.
The request is the central concept in relation to annotation work at Kognic. It connects a dataset and an annotation instruction, and the main bulk of progress monitoring and workforce management is done in relation to it.
Within the tab Annotation Requests, you find the request list, as shown in the image below. The list, in addition to giving you an overview of the project progress, acts as a navigation to the requests.
On top of the request list, you find controls that can help you refine the list for your goal. The controls are:
- Group by All the requests in a given project often share lots of common attributes, like taxonomy or input batch. This control allows you to see all requests grouped by their Taxonomy or Input Batch.
- Sort group by Sort the groups by their creation date, or alphabetically.
- Sort requests by Sort the requests within a group either by creation date or alphabetically.
- Filter Filter the requests by their Input Batch or their Taxonomy.
In the list, you can find all requests within the project. Each request card provides you insight into the request's progress as well as current task allocation (only available for Producing organizations).
Each card also acts as the main navigation to the request's own page - Request detailsο»Ώ. On the page request details, you can monitor and manage the request in greater detail.
In addition to general information, such as name and taxonomy, you find information about the request's current progress and state on the card. A description of this information follows below.
Statuses indicating the current phase of the request. They are:
Status | Name |
|---|---|
In Setup | The request requires more configuration |
Configured | The request is fully configured, and annotation can start. |
In Progress | Annotation started, i.e. any task in the request has been started |
Finished | All inputs have a final and Delivery Ready Annotation |
Inactive | The request is configured and some annotation is done but it got deactivated |
A progress bar indicating Delivery Ready, In-Progress and Total number of annotations to be done. The total number of annotations to be done equals the number of inputs that were uploaded to this Input Batch.
As a refresher (see Key conceptsο»Ώ), input is the entity to be annotated. For example, a single camera frame, a single camera + lidar frame but also a sequence of camera frames.
To view the numbers, hover over the progress bar and they will be displayed in a tooltip.
This column (and any other task-related information) is only visible for requests in which your organization has the role "Producer".
The information in this table gives you a snapshot of the current distribution of tasks among the task types and their completion status. This can help you understand how the production is progressing as well as where you might need to take action.
Task type | Description |
|---|---|
Annotate | The initial and usually the main (annotation) task inside the annotation workflow. |
Correct | Correct tasks allow edits on the annotation, with or without feedback as guidance. Depending on the workflow a Correct task follows either each Annotate task, randomly sampled Annotate tasks or such that got rejected during a Review/QA tasks. |
Review | Review tasks are similar to Correct tasks but provide additional functionality to give structured feedback on objects, specific timestamps, and specific sensors, inside the app. They also include a final judgment on whether to Accept or Reject the reviewed annotation |
Completion status | Description |
|---|---|
To Do | Tasks that are lined up to be done |
Ongoing | Tasks on which work has started |
Completed | Tasks that are submitted and done. |
As the interpretation of these numbers depends on the configured Annotation Workflow, an example reading of this data follows below.
Here is the right-most column of an example request.
This request has 248 inputs. The workflow follows an "Annotate - Review (- Correct)" pattern where each Input starts with an Annotate task, is followed by a Review task and, if that Review task is rejected, a correct task is issued until the following Review task is accepted.
- In the first column, we see that we completed 246 out of 248 Annotate tasks and 2 are still ongoing. Once those 2 are completed (and we don't upload more Inputs) the numbers in this first column are final and won't change anymore. This is because all workflows produce exactly 1 Annotate task per Input.
- In the second column, we see the Correct tasks. We immediately see that it's much less Correct tasks than Annotate tasks / Inputs, namely 102 in total. Currently 12 Correct tasks are ongoing and these are usually done by "Annotate" team members.
- In the third column, we see Review tasks. In total we got 307 Review tasks issued. This is more Review tasks than the 246 completed Annotate Task / Inputs which means that some Inputs already received two or more reviews, namely those where one or multiple Reviews were rejected and follow-up Reviews were issued based on the 61 completed Corrections.
- The total number of Reviews 307 matches the sum of 246 completed Annotate task and 61 completed Corrections.
- We got 194 Reviews in To Do and only 2 are Ongoing which is an indication that we maybe need more team members doing Reviews to work off that queue.
- The fact that we have 111 Reviews Completed and 102 Corrections in total means, in the case of this linear Annotate-Review-Correct workflow, that 102 Reviews were rejected. This doesn't necessarily map onto 102 Inputs. The annotations of some Inputs might have gotten rejected in several consecutive Reviews.
- From the progress bar (not screenshotted) we see that there's 9 Delivery-Ready Annotations. In the case of our current workflow, this means 9 of the completed Reviews were accepted. That fits the reading of Corrections from above, so from 111 Reviews 9 were accepted and 102 rejected. That's all Review metrics we can conclude at this point, for more details we need to wait for a more detailed view in our App.
The request is the main concept in the organization of annotation work at Kognic - and a lot of the daily monitoring and managing is done in relation to it. This is done from the request details page. You open it by clicking the request you want to access in the request list.
Read more about the request details page and its content in the section Request detailsο»Ώ
