Annotation Workflows

in the kognic platform, annotations can be produced using a selection of workflows these workflows determine the steps used to produce the annotations and in what order the steps should happen they define what type of steps (tasks) to complete before we have produced a deliverable annotation multiple factors, such as the desired quality, time, and cost, determine which workflow suits each request our available workflows two tier review phases annotate > full review > sampled review > delivery availability all organizations this request workflow features two quality assurance phases in the initial review phase, all inputs are selected for review and require reviewer acceptance to complete the phase the second review examines only a sample of inputs insights from this sample help estimate the quality of the remaining annotated inputs this approach allows for reduced review effort when the estimated quality is good, while still enabling full review when needed read more in two tier review docid\ nfxqay29jqu2ulbmqvpn7 two tier sampled review phases annotate > sampled review > sampled review > delivery availability all organizations this request workflow features two sampled review phases in these phases, only a subset of inputs is initially selected for review insights from this sample help estimate the quality of the remaining annotated inputs this approach allows for reduced review effort when the estimated quality is good, while still enabling full review when needed read more in two tier sampled review docid\ qlczrsolsxc2jud3uo6xn key concepts task a task is what ties an input to a team member and what type of action the team member should take with the input there are three types of tasks annotate , correct , and review on its way from original input to a delivery ready (or final ) annotation, the input spawns one or multiple tasks, sometimes even multiple tasks of the same type the number of tasks depends on the workflows used in the request and if tasks expire and get assigned to new users task types annotate an annotate task will be given to a team member with the purpose of them annotating it from scratch it is always the initial task inside the annotation workflow correct a task where the team member is asked to correct a task with existing annotations depending on the workflow, a correct task follows either each annotate task or tasks that got rejected during a review task review review tasks are similar to correct tasks but allow a team member to give structured feedback on objects, specific timestamps, and specific sensors inside the app review tasks also include a final judgement on whether to accept or reject the reviewed annotation task assignments for a user to be able to work on a task it has to be assigned to the user the user can now complete the assignment or let it expire once it expires , the task becomes available for assignment to other users the task is assigned either automatically by our system or manually by a manager, see team docid\ fiopd5ytqela7kyahrpyl for more details