--- title: Two-tier sampled review slug: two-tier-sampled-review icon: 🔍 docTags: createdAt: 2025-09-16T12:29:37.281Z --- This request workflow features two sampled review phases. In these phases, only a subset of inputs is initially selected for review. Insights from this sample help estimate the quality of the remaining annotated inputs. This approach allows for reduced review effort when the estimated quality is good, while still enabling full review when needed. You can change the selection probability of a Sampled Review phase (default: 20%) within [Phase Settings](https://docs.kognic.com/settings#3qmsT). ![](https://app.archbee.com/api/optimize/wLEz_RbWP6nIMpY-prjlm/f0HFJEfgDtavgZgBkXD9Y-20250916-124253.png "A general sketch of the workflow \"Two-tier sampled review\"") # Phase type: Annotate ::Image[]{src="https://api.archbee.com/api/optimize/wLEz_RbWP6nIMpY-prjlm/w1PWoXyF86MRfz7yqwjid_image.png" darkSrc="https://archbee-image-uploads.s3.amazonaws.com/wLEz_RbWP6nIMpY-prjlm/smhFSgWDl8H3-OMz182rM_annotationdark.png" size="70" width="2610" height="652" darkWidth="1305" darkHeight="326" position="center" caption="Flowchart of the phase type "Annotation"" showCaption="true"} In phases of the type "Annotation" all inputs that enter the phase will be annotated by a labeler. This is done in tasks with the action "Annotate". When a task is completed and the input has been annotated, it moves to the next phase. *** # Phase type: Sampled review :::hint{type="info"} This workflow has two sampled review phases placed after each other. Both have the same functionality but have separate settings, such as sampling level. ::: ![](https://api.archbee.com/api/optimize/wLEz_RbWP6nIMpY-prjlm/B1E3Tzi_a4ViiMRLDXE5J-20250916-130635.png "Flow chart of the phase type \"Sampled review\".") In phases of the type "Sampled review" only a number of the annotated inputs (intermediate annotations) that enter the phase are initially selected for review (approximately 20 % in the flowchart above). The inputs not selected for review stay in the phase until a manager decides, based on the result of the sampled review, if the non-selected inputs should be reviewed or if they can be sent directly to the next phase without being reviewed and potentially corrected. Annotated inputs selected for review (both initially selected and selected due to phase decision) must have their quality approved by a reviewer to complete and leave the phase. If the annotation quality isn't acceptable, a task with the action "correct" is created, and a labeler is asked to improve the quality according to the reviewer's feedback. The updated annotation is then reviewed again; if the quality isn't acceptable, the cycle is repeated until it is. :::hint{type="info"} An essential characteristic of the **Review Phase** is that it allows for giving structured **feedback** on the annotation. You can read more about this in the [ Feedback Tools](https://docs.kognic.com/AHwD-introduction) chapter. ::: :::ExpandableHeading ### Which user will get the correction task if a review is rejected? The correct task is assigned based on the type of the previous phase: - Previous phase was an annotate phase: the annotator gets the correct task. - Previous phase was a review phase and the input was reviewed there: the reviewer gets the correct task. - No eligible specific assignee (e.g., the input wasn't sampled in the previous sampled review phase, or the intended assignee doesn't have the matching `phase name: correct`stage: the task becomes available to all users with that workflow stage. The first to pick it up corrects it. ::: :::ExpandableHeading ### How is the sampling/selection done? When entering the phase, each annotated input has a set likelihood of being selected for review. That means if the sample size is set to 20 %, each annotated input entering the phase has a 20 % likelihood of being selected for review. Thus, the final sample size could add up to just below or just above the configured sample size. ::: :::ExpandableHeading ### What is the decision, and how does it work? ::Image[]{src="https://api.archbee.com/api/optimize/wLEz_RbWP6nIMpY-prjlm/4zmGuBqiAkMkuU-niyQmh_image.png" size="60" width="2976" height="976" position="center" caption="The decision interface" showCaption="true"} This phase contains a mandatory decision in which you, as a manager, based on the result of the initial review, determine what should happen to the annotated inputs that haven't been selected for initial review (incl. inputs that haven't entered the phase yet as well). 1. **Evaluate the review results** Start by evaluating the review results and consider if all annotated inputs require a review to ensure sufficient annotation quality. 2. **Decide if all annotated inputs should be reviewed or not ** If the sampled review results indicate that the annotated inputs are of sufficient quality when entering the phase, and a review and correction cycle isn't needed --> **Send the inputs not selected for review directly to the next phase.** If a review and correction cycle seems required to reach sufficient quality --> **Send all annotated inputs through review.** ::: :::hint{type="info"} Both of the available decisions only affect annotated inputs not selected for review. This means that inputs that have been selected for review need to be accepted by a reviewer to exit the phase. ::: :::hint{type="warning"} The decision is permanent and can't be reverted. ::: ## Configuring a sampled review as a client sampled review Often the people who need to sign off on annotation quality aren't internal reviewers. They're the client. A sampled review phase can be configured so the *client* gives the verdict instead of someone on the workforce. We call this a **client sampled review**. It's not a separate phase type. It's the same sampled review phase, with one extra setting that opens the review to client-side users. Sometimes the people who need to sign off on annotation quality aren't internal reviewers. They're the client. A sampled review phase can be configured so the *client* gives the verdict instead of someone on the workforce. We call this a **client sampled review**. It's not a separate phase type. It's the same sampled review phase, with one extra setting that opens the review to client-side users. ### When to use it A client sampled review fits when the client wants a final pass on annotations before delivery. If you need something heavier (quality metrics, formal verdicts, dispute handling, or rounds of client review), see [Client QA](docId\:WRC_OkYWzEq0kDYskB0mH) in the Kognic Standard Workflow. Client sampled review is the lightweight option; Client QA is the full one. ### How to enable it 1. Open the request and go to **Request Settings → Phases**. 2. Find the sampled review phase you want to convert. We recommend the last one before delivery. 3. Enable **Allow workspace/request owner and Client QA Contributors to manage the review stage**. That's it. The phase is now a client sampled review. Sampling rate and all other phase settings still work the same way. :::hint{type="warning"} **A note on naming.** You can't rename phases on a request, so the phase keeps the name it was given when the workflow was set up. In Request Settings, monitoring views, and task assignments it will look like any other sampled review unless its original name already reflects the intent. ::: ### What changes - **Who reviews:** users with the request organization role of *Owner* or *Client QA Contributor*. See [Request Access Management](docId\:ofZICvod6cGlj5sUvji6T) for how those roles work. Like internal reviewers, they need the matching workflow stage assigned (e.g., `phase name:review`) to be auto-assigned tasks. - **Who corrects rejected inputs:** the rule depends on the previous phase: Clients never receive correction tasks themselves. - Previous phase was an annotate phase: the annotator gets the correct task. - Previous phase was a review phase and the input was reviewed there: the reviewer gets the correct task. - No eligible specific assignee (e.g., the input wasn't sampled in the previous sampled review, or the intended assignee doesn't have the right workflow stage): the task becomes available to anyone with the matching `phase name: correct` stage. First to pick it up corrects it. - **Sampling:** unchanged. Sampling rate behaves the same as in a regular sampled review. Setting it to 100% is allowed and effectively turns the phase into a client *full* review. - **The manager decision:** unchanged. After the initial sampled results, a manager decides whether the unselected inputs go through review or move directly to delivery. The decision is permanent. ### Recommended placement Use the **last** sampled review phase before delivery. That way the client only sees annotations that have already passed at least one round of internal review. The platform doesn't enforce this. You *can* enable the setting on any sampled review phase, but other placements aren't recommended without a specific reason. *** # Phase type: Delivery When entering this last phase of the workflow, the annotated inputs become Delivery: Ready and have completed the request. 🎉 In the Delivery phase, there are more optional stages to iterate further on an annotation, if needed. These are "Delivery: Correct" and "Delivery: Review" [Read more about the different actions in this phase here.](https://docs.kognic.com/delivery)