Flexible Workflows

our flexible workflows consist of phases, which can be added, removed, or displayed in any order to suit your unique needs we currently have the following flexible workflows available for use, with more planned for the future available phases flexible workflows can consist of one or more of the following phases annotate full review sampled review delivery available workflows annotation + full review + sampled review this annotation workflow consists of three phases annotation, full review, and sampled review phase type annotate in phases of the type "annotation" all inputs that enter the phase will be annotated by a labeler this is done in tasks with the action "annotate" when a task is completed and the input has been annotated, it moves to the next phase phase type full review in phases of the type "full review" all of the intermediate annotations entering the phase are sent for review by a reviewer a reviewer must accept its quality for an annotated input to complete and leave the phase if the annotation quality isn't acceptable, a task with the action "correct" is created, and a labeler is asked to improve the quality according to the reviewer's feedback the updated annotation is then reviewed again; if the quality isn't acceptable, the cycle is repeated until it is an essential characteristic of the review phase is that it allows for giving structured feedback on the annotation you can read more about this in the feedback tools chapter 🙋🏻 which user will get the correction task if a review is rejected? we have configured this phase such that the team member who annotated the input in the previous phase is asked to correct the annotated input in this phase this team member must be assigned the corresponding workflow stage to ensure this works as intended i e , if this phase has been named "review 1" the team member needs to be assigned the workflow stage "review 1 correct" for us to assign this task directly to them if they haven't been assigned the correct workflow stage, another team member who has can access the task and start working on it phase type sampled review in phases of the type "sampled review" only a number of the annotated inputs (intermediate annotations) that enter the phase are initially selected for review (approximately 20 % in the flowchart above) the inputs not selected for review stay in the phase until a manager decides, based on the result of the sampled review, if the non selected inputs should be reviewed or if they can be sent directly to the next phase without being reviewed and potentially corrected annotated inputs selected for review (both initially selected and selected due to phase decision) must have their quality approved by a reviewer to complete and leave the phase if the annotation quality isn't acceptable, a task with the action "correct" is created, and a labeler is asked to improve the quality according to the reviewer's feedback the updated annotation is then reviewed again; if the quality isn't acceptable, the cycle is repeated until it is an essential characteristic of the review phase is that it allows for giving structured feedback on the annotation you can read more about this in the feedback tools chapter 🙋🏻 which user will get the correction task if a review is rejected? we have configured this phase such that the team member who reviewed the input in the previous phase (of the type "full review") is asked to correct the annotated input in this phase this team member must be assigned the corresponding workflow stage to ensure this works as intended i e , if this phase has been named "review 2" the team member needs to be assigned the workflow stage "review 2 correct" for us to assign this task directly to them if they haven't been assigned the correct workflow stage, another team member who has can access the task and start working on it 🙋🏻 how is the sampling/selection done? when entering the phase, each annotated input has a set likelihood of being selected for review that means if the sample size is set to 20 %, each annotated input entering the phase has a 20 % likelihood of being selected for review thus, the final sample size could add up to just below or just above the configured sample size 🙋🏻 what is the decision, and how does it work? this phase contains a mandatory decision in which you, as a manager, based on the result of the initial review, determine what should happen to the annotated inputs that haven't been selected for initial review (incl inputs that haven't entered the phase yet as well) evaluate the review results start by evaluating the review results and consider if all annotated inputs require a review to ensure sufficient annotation quality decide if all annotated inputs should be reviewed or not if the sampled review results indicate that the annotated inputs are of sufficient quality when entering the phase, and a review and correction cycle isn't needed > send the inputs not selected for review directly to the next phase if a review and correction cycle seems required to reach sufficient quality > send all annotated inputs through review both of the available decisions only affect annotated inputs not selected for review this means that inputs that have been selected for review need to be accepted by a reviewer to exit the phase the decision is permanent and can't be reverted phase type delivery when entering this last phase of the workflow, the annotated inputs become delivery ready and have completed the request 🎉 in the delivery phase, there are more optional stages to iterate further on an annotation, if needed these are "delivery correct" and "delivery review" read more about the different actions in this phase here annotation + sampled review + sampled review this annotation workflow consists of three phases annotation, sampled review, and sampled review you can change the selection probability of a sampled review (default 20%) within phase settings team management flexible workflows use a new way of assigning who does what type of task in the request read more about that in the chapter team 2 0 docid\ lvimgmdke cyq3syaf41j