ANNOTATION INTEGRATION
Review

5min
A review is a decision that determines whether an annotation is of sufficient quality or if it needs improvement. In order to facilitate efficient improvements, the person or system that made the decision can provide feedback﻿ data that details potential errors or mistakes. Please refer to our user documentation﻿ regarding feedback to gain insight on how these concepts are used in the platform, note that these describe the full set of possibilities and workflows for dealing with reviews and feedback - the possibilities enabled by this integration are currently somewhat limited, as detailed below.
Posting a review
The current integration capabilities only allow posting reviews for delivered annotations, and preliminary annotations in phases where you have review-access. The annotations are
identified using their open label UUID that can be found in the metadata (and in the file name) of downloaded annotations﻿.
When posting a review, the API expects feedback data as well as a boolean accepted that indicates whether the quality is perceived as sufficient or not. The API also expects an enum workflow that details how the improvement should happen. For a rejected review and the workflow 'correct', a single correction task is triggered, with no follow-up Review task. The completed Correction task becomes the new delivered Annotation.
A successful review post will return a UUID that identifies the review, and that can be used to fetch the posted feedback data. This will also create a task according to the chosen workflow. Depending on project set up, there might be a need for this to be coordinated with the project's managers.
Here's a example from kognic-io:
Python
def run(client: KognicIOClient, open_label_uuid: str, error_type_id: str) -> ReviewResponse:
    review = ReviewRequest(
        feedback_items=[
            AddFeedbackItem(
                sensor_id="<the id/name of the sensor>",
                frame_id="<the id of the frame>",  # in our OpenLabel file this is frame.frame_properties.external_id
                object_id="<the id of the object>",
                pin=AddFeedbackItemPin(x=0.0, y=0.0, z=0.0),
                description="I post this via the Python client",
                suggested_property=AddFeedbackItemSuggestedProperty(
                    property_name="propertyName", suggested_property_value="suggestedPropertyValue"
                ),
                error_type_id=error_type_id,
                metadata={"key": "value"},
            )
        ],
        workflow=ReviewWorkflowEnum.CORRECT,
        accepted=False,
    )

    return client.review.create_review(open_label_uuid=open_label_uuid, body=review)


if __name__ == "__main__":
    setup_logging(level="DEBUG")

    client = KognicIOClient()

    open_label_uuid = "<the uuid of the OpenLABEL>"
    error_type_id = "<the id of the error type>"
    annotation = run(client, open_label_uuid, error_type_id)
def run(client: KognicIOClient, open_label_uuid: str, error_type_id: str) -> ReviewResponse:
    review = ReviewRequest(
        feedback_items=[
            AddFeedbackItem(
                sensor_id="<the id/name of the sensor>",
                frame_id="<the id of the frame>",  # in our OpenLabel file this is frame.frame_properties.external_id
                object_id="<the id of the object>",
                pin=AddFeedbackItemPin(x=0.0, y=0.0, z=0.0),
                description="I post this via the Python client",
                suggested_property=AddFeedbackItemSuggestedProperty(
                    property_name="propertyName", suggested_property_value="suggestedPropertyValue"
                ),
                error_type_id=error_type_id,
                metadata={"key": "value"},
            )
        ],
        workflow=ReviewWorkflowEnum.CORRECT,
        accepted=False,
    )

    return client.review.create_review(open_label_uuid=open_label_uuid, body=review)


if __name__ == "__main__":
    setup_logging(level="DEBUG")

    client = KognicIOClient()

    open_label_uuid = "<the uuid of the OpenLABEL>"
    error_type_id = "<the id of the error type>"
    annotation = run(client, open_label_uuid, error_type_id)
﻿
Posting a partial review
A partial review is an incomplete review with optional feedback data, this capability may be used in order to achieve one or more of the following:
Prepare a review task with manual or machine generated feedback
Select what should be reviewed
This feature is available in requests where your organization is the producer﻿ or the owner﻿. 
When a partial review is posted, a review task will be created for the annotation. This review task will contain the feedback that may have been supplied when posting the partial review. The reviewer can then delete or invalidate any feedback they disagree with, or add their own feedback. It is currently not possible to override the workflow of a partial review, the default workflow is a review loop - where the correction of a rejected review task will be followed by a new review task. If the review is accepted by the reviewer, the annotation will be delivered again, updated to account for any potential changes the reviewer might have done.
Feedback
A feedback item details something noteworthy in an annotation. In order to do this the following information is
available to add, and will be available for annotators when improving the annotation:
Error type ID - a UUID selected from the list of available error types (fetch these using KognicIOClient().review.get_error_types())
Description - a string describing what should be improved
Sensor ID - the identifier of the sensor﻿ where the error appears
Frame ID - the identifier of the frame﻿ the error appears. In our OpenLabel file this is frame.frame_properties.external_id
Object ID - the identifier of a particular object that is subject to this particular feedback
Pin - a pointer to a specific area where something is of interest, this can be used to indicate missing objects (a
pin contains mandatory x and y coordinates with an optional z coordinate, the coordinate unit in images are pixels and
is sensor specific in 3d data)
Suggested properties - when inappropriate property values are discovered, this field can be used to indicate a more
appropriate value
Metadata - this field may contain any data, it could be used to identify the version of the system that generated the
feedback
﻿
Introduction