Annotation Requests

under project management annotation requests, you can find and navigate to non archived annotation requests, by project the request is the central concept in relation to annotation work at kognic it connects a dataset and an annotation instruction, and the main bulk of progress monitoring and workforce management is done in relation to it annotation request list within the tab annotation requests , you find the request list, as shown in the image below the list, in addition to giving you an overview of the project progress , acts as a navigation to the requests list controls on top of the request list, you find controls that can help you refine the list for your goal the controls are group by all the requests in a given project often share lots of common attributes, like taxonomy or input batch this control allows you to see all requests grouped by their taxonomy or input batch sort group by sort the groups by their creation date, or alphabetically sort requests by sort the requests within a group either by creation date or alphabetically filter filter the requests by their input batch or their taxonomy the annotation request list in the list, you can find all requests within the project each request card provides you insight into the request's progress as well as current task allocation (only available for producing organizations) each card also acts as the main navigation to the request's own page request details docid\ wayww2ekb81abyamgbkac on the page request details, you can monitor and manage the request in greater detail in addition to general information, such as name and taxonomy, you find information about the request's current progress and state on the card a description of this information follows below status statuses indicating the current phase of the request they are status name in setup the request requires more configuration configured the request is fully configured, and annotation can start in progress annotation started, i e any task in the request has been started finished all inputs have a final and delivery ready annotation inactive the request is configured and some annotation is done but it got deactivated delivery ready annotations a progress bar indicating delivery ready , in progress and total number of annotations to be done the total number of annotations to be done equals the number of inputs that were uploaded to this input batch as a refresher (see key concepts docid\ f4wsr2qmqwx5nd6ugxwuq ), input is the entity to be annotated for example, a single camera frame, a single camera + lidar frame but also a sequence of camera frames to view the numbers, hover over the progress bar and they will be displayed in a tooltip task status this column (and any other task related information) is only visible for requests in which your organization has the role "producer" the information in this table gives you a snapshot of the current distribution of tasks among the task types and their completion status this can help you understand how the production is progressing as well as where you might need to take action task type description annotate the initial and usually the main (annotation) task inside the annotation workflow correct correct tasks allow edits on the annotation, with or without feedback as guidance depending on the workflow a correct task follows either each annotate task, randomly sampled annotate tasks or such that got rejected during a review/qa tasks review review tasks are similar to correct tasks but provide additional functionality to give structured feedback on objects, specific timestamps, and specific sensors, inside the app they also include a final judgment on whether to accept or reject the reviewed annotation completion status description t o do tasks that are lined up to be done o ngoing tasks on which work has started c ompleted tasks that are submitted and done example reading of task status table as the interpretation of these numbers depends on the configured annotation workflow, an example reading of this data follows below here is the right most column of an example request this request has 248 inputs the workflow follows an "annotate review ( correct)" pattern where each input starts with an annotate task, is followed by a review task and, if that review task is rejected, a correct task is issued until the following review task is accepted in the first column , we see that we completed 246 out of 248 annotate tasks and 2 are still ongoing once those 2 are completed (and we don't upload more inputs) the numbers in this first column are final and won't change anymore this is because all workflows produce exactly 1 annotate task per input in the second column, we see the correct tasks we immediately see that it's much less correct tasks than annotate tasks / inputs, namely 102 in total currently 12 correct tasks are ongoing and these are usually done by "annotate" team members in the third column, we see review tasks in total we got 307 review tasks issued this is more review tasks than the 246 completed annotate task / inputs which means that some inputs already received two or more reviews, namely those where one or multiple reviews were rejected and follow up reviews were issued based on the 61 completed corrections the total number of reviews 307 matches the sum of 246 completed annotate task and 61 completed corrections we got 194 reviews in to do and only 2 are ongoing which is an indication that we maybe need more team members doing reviews to work off that queue the fact that we have 111 reviews completed and 102 corrections in total means, in the case of this linear annotate review correct workflow, that 102 reviews were rejected this doesn't necessarily map onto 102 inputs the annotations of some inputs might have gotten rejected in several consecutive reviews from the progress bar (not screenshotted) we see that there's 9 delivery ready annotations in the case of our current workflow, this means 9 of the completed reviews were accepted that fits the reading of corrections from above, so from 111 reviews 9 were accepted and 102 rejected that's all review metrics we can conclude at this point, for more details we need to wait for a more detailed view in our app view and manage a request the request is the main concept in the organization of annotation work at kognic and a lot of the daily monitoring and managing is done in relation to it this is done from the request details page you open it by clicking the request you want to access in the request list read more about the request details page and its content in the section request details docid\ wayww2ekb81abyamgbkac