Annotation Points and Target

knowing how much data is being produced, and how productive annotators are, are important measures for running a project there are several metrics which are common in the industry, such as number of shapes or objects, the amount of properties set or tasks completed to normalize these measures across different projects and taxonomies kognic has developed a suite of productivity tools the basis for these tools is the annotation points specification and the annotation points target finding the configuration menu to setup annotation points and targets visit project settings , found under project management from there you should be able to see the annotation points tab select it and now you're able to use the annotation points configuration annotation points tab connection with the annotation instruction to be able to use the annotation points configuration, you need to have configured an annotation instruction for the project the annotation points configuration uses all the annotation instruction revisions that have been made during the project's lifecycle that means that you might see fields that don't exist within the currently used annotation instruction this is perfectly fine, and to be expected an ai allows for possibly multiple of the same geometries if you’ve configured an annotation instruction to use both 2d semantic segmentation, and 2d instance segmentation without having allow instances with multiple polygons checked then both of these will be merged into 2d segmentation annotation points specification each article that can be created in the kognic platform can be given an amount of annotation points that will be awarded the annotator adding them in a task these articles can be things like completing the task, finishing an object or setting properties below is an example of how a specification is applied to some annotator work to amount to a number of annotation points specification task 10p cuboid object 4p 2d bounding box object 2p task content cuboid objects 10 2d bounding box objects 7 annotation points = 10p 1 + 4p 10 + 2p 7 = 10 + 40 + 14 = 64p annotation points target to measure how productive an annotator is there needs to be a target which they are expected to reach the unit is annotation points per hour configuration the configuration allows for you to set the following values which then make up an entire annotation points specification annotation point target this is the total value of expected annotation points that the annotators are supposed to be meeting every hour annotation points target points per task this is how many annotation points a task should be given to the annotators upon submission of the task points per task points per geometry this is where you’re able to configure how many annotation points that should be rewarded for a specific geometry the fields that you’re able to set the values for are base points per object this is how many points an object should be rewarded additional points per shape how many points should be awarded per shape (object frames that the object lives in) points per geometry points per property you’re able to configure points per property for the object properties or scene properties that exist within the ai the values for dynamic properties are those that may change over time such as `occlusion`, whereas consistent properties are those that are consistent such as car points per property review point target similar to the annotation points target, this setting defines the expected productivity for reviewers the unit is reviewed annotation points per hour setting this target allows the system to calculate a review budget (in hours) for every task this provides reviewers with a clear time expectation based on the complexity of the annotation work specify a review point target review budget calculation \text{review budget (hours)} = \frac{\text{total task annotation points}}{\text{review target (ap/hour)}} example task value 2000 annotation points review target 1000 ap/hour result the reviewer has a 2 hour budget to review the task review 1 only time budgets are calculated for the first review round only (review 2+ are excluded) expected learning curve the expected learning curve defines the speed ramp up you expect from new annotators during their first weeks in a project it appears as a shaded band on the learning curve metrics chart, giving project managers a visual reference for whether annotators are onboarding on track this section is only available when an annotation points target has been configured configuration each row represents one onboarding week (starting from week 0) and specifies a min and max value as a percentage of the annotation speed target min — the lower bound of expected speed for that week max — the upper bound of expected speed for that week for example, if the annotation speed target is 100 ap/hour and week 1 is configured as 40%–60%, the expected speed range for that week is 40–60 ap/hour you can add up to 7 weeks (w0–w6) the last defined week's values automatically extend to fill any remaining weeks beyond the configured range saving click save learning curve to persist the configuration changes are reflected immediately on the learning curve metrics tab