---
title: Review
slug: api-guide/review
docTags: 
createdAt: 2024-12-16T13:32:55.569Z
---

A review is a decision that determines whether an annotation is of sufficient quality or if it needs improvement. In order to facilitate efficient improvements, the person or system that made the decision can provide [feedback](docId\:EAokNIu0RtITvVuDdsGVA) data that details potential errors or mistakes. Please refer to our [user documentation](docId\:kkOaaDN2hL33jbtqug7Po) regarding feedback to gain insight on how these concepts are used in the platform, note that these describe the full set of possibilities and workflows for dealing with reviews and feedback - the possibilities enabled by this integration are currently somewhat limited, as detailed below.

## Posting a review

The current integration capabilities only allow posting reviews for delivered annotations, and preliminary annotations in phases where you have review-access. The annotations are
identified using their open label UUID that can be found in the metadata (and in the file name) of [downloaded annotations](docId:5J-McgJYo5xFFxqkKrQGo).

When posting a review, the API expects feedback data as well as a boolean `accepted` that indicates whether the quality is perceived as sufficient or not. The API also expects an enum `workflow` that details how the improvement should happen. For a rejected review and the workflow 'correct', a single correction task is triggered, with no follow-up Review task. The completed Correction task becomes the new delivered Annotation.

A successful review post will return a UUID that identifies the review, and that can be used to fetch the posted feedback data. This will also create a task according to the chosen workflow. Depending on project set up, there might be a need for this to be coordinated with the project's managers.

Here's a example from kognic-io:

```python
def run(client: KognicIOClient, open_label_uuid: str, error_type_id: str) -> ReviewResponse:
    review = ReviewRequest(
        feedback_items=[
            AddFeedbackItem(
                sensor_id="<the id/name of the sensor>",
                frame_id="<the id of the frame>",  # in our OpenLabel file this is frame.frame_properties.external_id
                object_id="<the id of the object>",
                pin=AddFeedbackItemPin(x=0.0, y=0.0, z=0.0),
                description="I post this via the Python client",
                suggested_property=AddFeedbackItemSuggestedProperty(
                    property_name="propertyName", suggested_property_value="suggestedPropertyValue"
                ),
                error_type_id=error_type_id,
                metadata={"key": "value"},
            )
        ],
        workflow=ReviewWorkflowEnum.CORRECT,
        accepted=False,
    )

    return client.review.create_review(open_label_uuid=open_label_uuid, body=review)


if __name__ == "__main__":
    setup_logging(level="DEBUG")

    client = KognicIOClient()

    open_label_uuid = "<the uuid of the OpenLABEL>"
    error_type_id = "<the id of the error type>"
    annotation = run(client, open_label_uuid, error_type_id)
```

## Posting a partial review

A partial review is an incomplete review with optional feedback data, this capability may be used in order to achieve one or more of the following:

- Prepare a review task with manual or machine generated feedback
- Select what should be reviewed

This feature is available in requests where your organization is the [producer](docId\:wayWw2ekB81abYaMGbkaC) or the [owner](docId\:wayWw2ekB81abYaMGbkaC).&#x20;

When a partial review is posted, a review task will be created for the annotation. This review task will contain the feedback that may have been supplied when posting the partial review. The reviewer can then delete or invalidate any feedback they disagree with, or add their own feedback. It is currently not possible to override the workflow of a partial review, the default workflow is a review loop - where the correction of a rejected review task will be followed by a new review task. If the review is accepted by the reviewer, the annotation will be delivered again, updated to account for any potential changes the reviewer might have done.

## Feedback

A feedback item details something noteworthy in an annotation. In order to do this the following information is
available to add, and will be available for annotators when improving the annotation:

- **Error type ID** - a UUID selected from the list of available error types (fetch these using `KognicIOClient().review.get_error_types()`)
- **Description** - a string describing what should be improved
- **Sensor ID** - the identifier of the [sensor](docId\:yUnPNPwUhZlGG9Wb9Qnk8) where the error appears
- **Frame ID** - the identifier of the [frame](docId\:yUnPNPwUhZlGG9Wb9Qnk8) the error appears. In our OpenLabel file this is `frame.frame_properties.external_id`
- **Object ID** - the identifier of a particular object that is subject to this particular feedback
- **Pin** - a pointer to a specific area where something is of interest, this can be used to indicate missing objects (a
  pin contains mandatory x and y coordinates with an optional z coordinate, the coordinate unit in images are pixels and
  is sensor specific in 3d data)
- **Suggested properties** - when inappropriate property values are discovered, this field can be used to indicate a more
  appropriate value
- **Metadata** - this field may contain any data, it could be used to identify the version of the system that generated the
  feedback