Delivery

what is the delivery phase? every workflow ends in a delivery phase when an input enters this phase, it becomes a delivery ready annotation and completes the request 🎉 available actions in the delivery phase, there are multiple ways to iterate further on an annotation, if needed these are presented below send for correction managers inside a requests producer organization of a request you can send a delivery ready input for correction this will move the input to the workflow stage delivery correct and create a correction task the action will remove the input from delivery ready state but keep it in the same workflow phase (delivery) if desired, you can assign the correction task to a specific workforce member however, this is an optional function and not selecting this means the task will be available for workforce members configured to work in the stage delivery correct send for review managers of a request's producer or owner organizations can create review tasks for delivery ready inputs just after sending a delivery ready input for review and while the input is in its first review round it will be part of both the workflow stages delivery review and delivery ready this is because the delivery ready annotation isn't replaced until it has been accepted or rejected in review if desired, you can assign the review task to a specific workforce member however, this is an optional function and not selecting this means the task will be available for workforce members configured to work in delivery review quick accept accepts and moves inputs with unstarted review tasks to the stage delivery ready without needing to be accepted by a reviewer inside a review task inputs with unstarted review tasks have the task type "review" and current task state "to do" find the input/inputs you want to quick accept inside the phase inputs table you can use the search bar or the input id filter if needed if you only want to quick accept one input, you can use the "quick accept" option in the inputs options menu if you want to quick accept multiple inputs, select them using the row checkboxes and then click the "quick accept" option inside the action bar that appears at the bottom of the page download annotation for delivery ready inputs, you can download the annotation in the openlabel format you do this by opening the inputs option menu and clicking the button "download annotation" available monitoring progress this section's content helps you understand how many inputs are delivery ready and whether any additional iteration is being performed inside the delivery phase inputs in this phase the number of inputs inside the delivery phase that currently are in the stages delivery review and delivery correct inputs ready for delivery the percentage out of all inputs that are ready for delivery and download general phase progress in this graph, you get insight into how the requests inputs are distributed in relation to this phase and its workflow stages you can see how many inputs are in an earlier phase, how many are inside the phase, and in which workflow stage they are, as well as how many have are delivery ready phase inputs in the phase input table, you can see all inputs that are currently inside the delivery phase for each input, you can see when it changed workflow stage, how many tasks have been done on it in the current phase, and the state and type of any current task the actions send for review , send for correction , and quick accept are available for specific inputs you can read more about them in the section "available actions" above error summary with the error summary you get insight into what issues reviewers have found and commented on during the delivery phase's review tasks it helps you understand the most common and less frequent identified issues the error summary insights are based on feedback items written by the phase's reviewers absolute numbers represent actual feedback items, not the edits made in response to the feedback no of correction requests the number of feedback items of the type "correction requests" this is the sum of all errors shown in the "error type distribution" to the right no of feedback items the number of feedback items of the type "advice" these are excluded from the chart "error type distribution" and "suggested properties" error type distribution shows the absolute count and relative share of all feedback items categorized as "correction requests", grouped by their error type suggested properties for those items with the error type properties, this shows the distribution of properties that we affected each error indicated as properties has a single property connected to it individual feedback items this section helps you to get an overview of all given feedback, to answer questions such as how detailed and critical is the feedback of my colleague reviewers? are the reviewers giving valid feedback given the current guideline? what is feedback where the reviewer and annotator are discussing in the comments? how does feedback of type "missingobject" look? what type of feedback is marked as "invalid"? the items are split up by their feedback type correction requests in this section, you see feedback items for the type correction request these are things that the reviewer wants to get corrected before accepting the review you can filter the feedback items by their resolved status, the error type, whether a discussion thread exists, or whether the overall review of the input has been accepted yet below is a description of what information is available for each correction request status status comment unresolved when created by the reviewer, and not yet approved corrected when the annotator has fixed the issue mentioned in the correction request resolved when the reviewer has approved the annotators fix invalid an item can be marked as "invalid" by the user if they think it's not accurate with respect to the guidelines of the task, this can be due to mistakes in machine generated feedback or from human reviewers an item can be "unresolved" even if the overall review was accepted, or the other way around error type the type of error that was selected in the correction request suggested property if the error type is "properties", this column shows which property and value was suggested by the reviewer comment shows the description that the reviewer might have given thread exists will say "yes", if there was any reply to the item, i e a discussion thread has been started in relation to the item external scene id the scene id of the reviewed annotation current round the review round in which the input of this feedback item currently is all inputs start in round 1 with each rejected review, they progress 1 round forward accepted review whether the overall review was accepted or not feedback in this section you see feedback items of the type advice as the underlying data has less structure, the table has fewer columns and filtering options, but otherwise, it looks the same as the one above