Pre-annotations

reindex scenepre annotations have many uses in ground truth production the pre annotations feature allows information about the objects already known to be present in an input to be specified please reach out to our advisory services team to see how they can best be used for your use case the kognic platform supports uploading pre annotations in the openlabel format using the kognic openlabel package https //pypi org/project/kognic openlabel creating pre annotations using the kognic io client there are 3 steps that are needed in order to create pre annotations in the kognic platform create a scene by uploading all the needed data upload an openlabel annotation as a pre annotation create an input from the scene note that these steps can be performed in one call with the create inputs function, see working with scenes & inputs docid\ utyshj bng1v85n2e6 eh 1\ creating a scene start by creating a scene \# create scene but not input since we don't provide project or batch scene response = client lidars and cameras sequence create( lidars and cameras seq, dryrun=dryrun ) note that you now have to wait for the scene to be created before you can proceed to the next step more information this can be found working with scenes & inputs docid\ utyshj bng1v85n2e6 eh 2\ uploading an openlabel annotation the pre annotation can be uploaded to the kognic platform once the scene has been created successfully load your openlabel annotation according to the documentation in kognic openlabel and upload it to the kognic platform as such client pre annotation create( scene uuid=scene response scene uuid, # from step 1 pre annotation=openlabelannotation( ), dryrun=dryrun ) 3\ create input when the scene and pre annotation have been successfully created, the input can be created this will add it to the latest open batch in a project, or the specific batch that's specified, and be ready for annotation with the pre annotation present client lidars and cameras sequence create from scene( scene uuid=scene response scene uuid, # from step 1 project=project, # important this is the external id and not the title dryrun=dryrun ) openlabel support pre annotations use the openlabel format/schema but not all openlabel features are supported in pre annotations unsupported pre annotation features these features or combinations of features are not currently supported, or only have partial support static geometries not supported these are bounding boxes, cuboids, etc declared in the openlabel under objects objectdata geometry specific attributes not supported on 3d geometry these are attributes declared in the openlabel on a single geometric shape, in other words an attribute that only applies to the object as seen by one sensor; a common example is occlusion which is recorded separately for each camera may also be referred to as source , stream or sensor specific attributes 3d geometry is anything that can be drawn when annotating a pointcloud, e g cuboids geometry specific attributes are permitted on 2d geometry e g bounding boxes note that the key concepts docid 14dcxcasr7o nam1xyu5z , must designate a property as source specific before it may be used in this way the stream attribute is a special case and is excepted from this rule supported pre annotation features geometries objects cannot have multiple 3d geometries in the same frame name openlabel field description attributes cuboid cuboid cuboid in 3d bounding box bbox bounding box in 2d 3d line poly3d line in 3d append the first point at the end if you want it to be closed polygon poly2d openlabel format docid\ awhhsrojm96zowliryw7d in 2d is hole multi polygon poly2d openlabel format docid\ awhhsrojm96zowliryw7d in 2d is hole & polygon id curve poly2d openlabel format docid\ awhhsrojm96zowliryw7d or line in 2d interpolation method 2d point point2d openlabel format docid\ awhhsrojm96zowliryw7d group of 2d points point2d openlabel format docid\ awhhsrojm96zowliryw7d point class 3d semantic segmentation binary 3d instance segmentation binary 3d point point3d 3d lane poly3d built by providing 2 poly3d , one for each edge of the lane specify which lane edge using the lane edge attribute, one with the value right and one with the value left this geometry therefore is an exception to the rule that objects can't have multiple 3d geometries in the same frame lane edge note that all geometries should be specified under frames rather than in the root of the pre annotation 3d geometries should be expressed in the lidar coordinate system in the single lidar case, but in the reference coordinate system in the multi lidar case the rotation of cuboids should be the same as that in openlabel format docid\ awhhsrojm96zowliryw7d 2d geometries should be expressed in pixel coordinates see coordinate systems docid\ xqf6uaqsofwvavmtepdec for more information attributes text num boolean for 2d geometry, attributes may be specified as geometry specific (aka source/sensor specific), or object specific attributes can be static (specified in the objects key) or dynamic (specified in the object data for the object in the frame) and must be allowed by the key concepts docid 14dcxcasr7o nam1xyu5z , if one exists geometry specific attributes (those which appear on a single shape within frames) must also be declared as such in the task definition; arbitrary properties cannot be used in a source specific way contexts context is used to define scene properties in the kognic platform each context contains one property value contexts comes in four modes static global the value is valid for all frames and sensor of the scene dynamic global the value is valid for all sensor, but may change across frames static source specific the value is valid for all frames but only for 1 specific stream dynamic source specific the value is valid for only 1 specific stream and may change across frames we will be referring to two types of context object context object (or simply context) defined under the root key contexts frame context object (or simply frame context) defined in a frame all contexts used in the openlabel have to be defined under the context key in the root of the openlabel at kognic the key of the context is only used to reference a context when used in other places in the openlabel (e g in a frame) and we set it to a string of incrementing numbers the context type will correspond to property name in the kognic app and the value of the property is set in the attributes of the context if a source specific property should exist in multiple sensors, e g each camera sensor has a boolean property "sees car", then there will be one context object per sensor all with the same context type for contexts we support the following attribute types text num boolean vec only string vecs context mode has stream attribute in context has non stream attribute in context has attribute in frame contex has more than 1 non stream attribute in context has more than 1 attribute in frame contex static global ❌ ✅ ❌ ❌ ❌ dynamic global ❌ ❌ ✅ ❌ ❌ static source specific ✅ ✅ ❌ ❌ ❌ dynamic source specific ✅ ❌ ✅ ❌ ❌ see examples pre annotations docid\ s1enygschymkq7jff dcx frames every pre annotation must contain frames with unique timestamps that are among the ones specified in the scene the reason for this is that the timestamps are used to map the frame in the pre annotation to the correct frame in the scene in the static case, one frame should be used with timestamp 0 relations supported as per our documentation for the open label format source specific relations are not supported note that geometry collection docid\ b slhzbxrpygxqvdytazd is a special case of a relationship, using the type geometry collection streams every geometry must have the stream property specified this property determines which stream (or sensor) that the geometry appears in it is important that the stream is among the ones specified in the scene and of the same type, for example camera or lidar sparseness pre annotations can be sparse, meaning that its objects or geometries do not need to be present in every frame instead, they can be present in a subset of frames and then interpolated in the frames in between utilizing this feature can speed up the annotation process significantly for sequences sparseness can be accomplished in two different ways, either by using object data pointers or the boolean property interpolated the former is the recommended way of doing it in most cases since it will lead to a more compact pre annotation the latter is useful when the pre annotation is created from exported annotations from the kognic platform interpolation is done by linearly interpolating the geometry values between key frames this is done in pixel coordinates for 2d geometries for 3d geometries, the interpolation can be done in either the frame local coordinate system or the world coordinate system (see coordinate systems docid\ xqf6uaqsofwvavmtepdec ) this is configured in the annotation instruction so reach out to the kognic team about this if you are unsure note that interpolation in the world coordinate system is recommended but requires that the scene contains ego poses object data pointers in openlabel, object data pointers are used to create a specification for objects for example, you can specify what attributes and geometries that are used for specific objects in addition, you can specify for which frames that these are present if a geometry is specified in the object data pointer, it will be present in all frames that the object data pointer is pointing to if the geometry is not provided in some of these frames, it will be interpolated note that geometries must be provided for the first and last frame in the object data pointer otherwise, the pre annotation will be rejected one limitation is that a geometry must be in the same stream for all frames when using object data pointers this is because interpolation is done in the stream coordinate system if you need to use geometries of the same type in different streams, you can simply use different names for the geometries in the different streams pre annotations docid\ s1enygschymkq7jff dcx shows an example of how to use object data pointers interpolated property the boolean property interpolated can be used to specify that a geometry should be interpolated geometries are still required to be present in interpolated frames but their geometry values will be ignored note that interpolated geometries must have corresponding geometry (interpolated or not) in the first of the pre annotation otherwise, the pre annotation will be rejected using the interpolated property is the recommended way of doing it when the pre annotation is created from exported annotations from the kognic platform pre annotations docid\ s1enygschymkq7jff dcx shows an example of how to use the interpolated property attributes attributes are handled differently compared to geometries if an attribute is not present in a frame, its last value will simply be used if the object (or geometry if the property is source specific) is present in the frame if the object is not present in the frame, the attribute will be ignored dense attributes will be sparsified automatically when the pre annotation is uploaded to the kognic platform kognic reserved object properties there are certain properties that can be set on an object to toggle various behavior in the kognic platform locked geometries if an object and its geometries in the pre annotation is already of sufficient quality, or should remain unchanged during use of the pre annotation, you can mark it as locked the lock is put on an object level, and will affect all the objects geometries { "openlabel" { "objects" { "object uuid" { "name" "object uuid", "object data" { "boolean" \[ { "name" "kognic locked geometries", "val" true } ] }, "object data pointers" {}, "type" "vehicle" } } } }import kognic openlabel models as olm uuid1 = str(uuid uuid4()) object = olm object( name=uuid1, type="car", object data=olm objectdata( boolean=\[ olm boolean(name="kognic locked geometries", val=true), ] ), ) openlabel = olm openlabel(objects={uuid1 object}, metadata=olm metadata(schema version="1 0 0")) openlabel annotation = olm openlabelannotation(openlabel=openlabel) stationary objects a stationary object is something that can move, but doesn't a good example of this is a parked car this is different from a static object, which can't move, such as a landmark objects can be marked as stationary to enable certain platform features { "openlabel" { "objects" { "object uuid" { "name" "object uuid", "object data" { "boolean" \[ { "name" "kognic stationary object", "val" true } ] }, "object data pointers" {}, "type" "vehicle" } } } }import kognic openlabel models as olm uuid1 = str(uuid uuid4()) object = olm object( name=uuid1, type="car", object data=olm objectdata( boolean=\[ olm boolean(name="kognic stationary object", val=true), ] ), ) openlabel = olm openlabel(objects={uuid1 object}, metadata=olm metadata(schema version="1 0 0")) openlabel annotation = olm openlabelannotation(openlabel=openlabel) examples below follows examples of supported pre annotations 3d cuboid and 2d bounding box with a static property { "openlabel" { "frame intervals" \[], "frames" { "0" { "frame properties" { "timestamp" 0, "external id" "0", "streams" { "lidar1" {}, "zfc" {} } }, "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "object data" { "bbox" \[ { "attributes" { "text" \[{ "name" "stream", "val" "zfc" }] }, "name" "bounding box 1", "val" \[1 0, 1 0, 40 0, 30 0] } ], "cuboid" \[ { "attributes" { "text" \[{ "name" "stream", "val" "lidar1" }] }, "name" "cuboid 89ac8a2b", "val" \[ 2 079312801361084, 18 919870376586914, 0 3359137773513794, 0 002808041640852679, 0 022641949116037438, 0 06772797660868829, 0 9974429197838155, 1 767102435869269, 4 099334155319101, 1 3691029802958168 ] } ] } } } } }, "metadata" { "schema version" "1 0 0" }, "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "name" "1232b4f4 e3ca 446a 91cb d8d403703df7", "object data" { "text" \[{ "name" "color", "val" "red" }] }, "type" "passengercar" } }, "streams" { "lidar1" { "type" "lidar" }, "zfc" { "type" "camera" } } } } 3d line with a dynamic property { "openlabel" { "frame intervals" \[{ "frame end" 0, "frame start" 0 }], "frames" { "0" { "frame properties" { "streams" { "lidar" {} }, "timestamp" 0, "external id" "0" }, "objects" { "cc06aced d7dc 4638 a6e9 dc7f5e215340" { "object data" { "poly3d" \[ { "attributes" { "text" \[{ "name" "stream", "val" "lidar" }] }, "closed" false, "name" "line 3d 1", "val" \[ 5 0, 0 0, 0 0, 5 0, 10 0, 0 0, 5 0, 10 0, 0 0, 5 0, 0 0, 0 0, 5 0, 0 0, 0 0 ] } ], "text" \[{ "name" "occluded", "val" "no" }] } } } } }, "metadata" { "schema version" "1 0 0" }, "objects" { "cc06aced d7dc 4638 a6e9 dc7f5e215340" { "name" "cc06aced", "type" "region" } }, "streams" { "lidar" { "type" "lidar" }, "zfc" { "type" "camera" } } } } sparseness with object data pointers in the example below the object 1232b4f4 e3ca 446a 91cb d8d403703df7 has a bounding box called the bbox name that is provided in frames 0 and 3 in frames 1 and 2, the bounding box will be interpolated { "openlabel" { "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "name" "car name", "type" "car", "object data pointers" { "the bbox name" { "type" "bbox", "frame intervals" \[{"frame start" 0, "frame end" 3}] } } } }, "frames" { "0" { , "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "object data" { "bbox" \[{"name" "the bbox name", }] } } } }, "1" {}, "2" {}, "3" { , "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "object data" { "bbox" \[{"name" "the bbox name", }] } } } } } } } sparseness with interpolated property in the example below sparseness is determined using the interpolated property the object 1232b4f4 e3ca 446a 91cb d8d403703df7 has a bounding box for which the interpolated property is set to true in frames 1 and 2 but not in frames 0 and 3 the geometry values in frames 1 and 2 are ignored and instead interpolated from the geometry values in frames 0 and 3 { "openlabel" { , "frames" { "0" { , "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "object data" { "bbox" \[ { "attributes" { "stream" \[{ "name" "stream", "val" "cam" }], "boolean" \[{ "name" "interpolated", "val" false }] }, } ] } } } }, "1" { , "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "object data" { "bbox" \[ { "attributes" { "stream" \[{ "name" "stream", "val" "cam" }], "boolean" \[{ "name" "interpolated", "val" true }] }, } ] } } } }, "2" { , "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "object data" { "bbox" \[ { "attributes" { "stream" \[{ "name" "stream", "val" "cam" }], "boolean" \[{ "name" "interpolated", "val" true }] }, } ] } } } }, "3" { , "objects" { "1232b4f4 e3ca 446a 91cb d8d403703df7" { "object data" { "bbox" \[ { "attributes" { "stream" \[{ "name" "stream", "val" "cam" }], "boolean" \[{ "name" "interpolated", "val" true }] }, } ] } } } } } } } openlabel with context { "openlabel" { "context" { "0" { // a global static context "name" "", // this has no meaning to us "type" "isdaytime", "attributes" { "boolean" \[{ "name" "", // this has no meaning to us "val" true }] } }, "1" { // a global dynamic context "name" "", "type" "numberofsignspassed", "attributes" {} }, "2" { // a stream specific static context "name" "", "type" "iscamerapointingforward", "attributes" { "text" \[{ "name" "stream", "value" "cam1"}], "boolean" \[{ "name" "", "value" true} } }, "3" { // same property as above but in a different sensor "name" "", "type" "iscamerapointingforward", "attributes" { "text" \[{ "name" "stream", "value" "cam2"}], "boolean" \[{ "name" "", "value" true} } }, "4" { // a stream specific dynamic context "name" "", "type" "landscapecharacteristics" "attributes" { "text" \[{ "name" "stream", "value" "cam1"}] } } }, "frames" { "0" { , "contexts" { "1" { "attributes" { "num" \[{ "name" "", "value" 7 }] } } } }, "1" {}, "2" {}, "3" { , "objects" { "4" { "attributes" { "vec" \[{ "name" "", "value" \["trees", "fields"] }] } } } } } } } 3d semantic and instance segmentation when uploading pre annotations for 3d segmentation tasks the openlabel contains both the classifications as well as any instances that are present in the scene the classification is contained by a special object with the object type 3dpointcloudsegmentation this object must contain a binary object data entry with the classifications encoded using rle any instances should have an object entry with a classification id value kognic imposes a classification numbering scheme as follows range meaning 0 unclassified 1 255 semantic classification, e g road, building 256 65535 instance classification, e g car1, car2, pedestrian1, pedestrian2 { "openlabel" { "frames" { "0" { "objects" { "ecdb280e 0ff0 4f59 b9f2 9135e42f991c" { "object data" {} }, "b088dcbf c7fe 46a5 a71d 88ef49bdc107" { "object data" { "binary" \[ { "attributes" { "text" \[ { "val" "lidar", "name" "stream" } ] }, "data type" "", "encoding" "rle", "name" "labels", "val" "\<rle string>" } ] } } } } }, "metadata" { "schema version" "1 0 0" }, "objects" { "ecdb280e 0ff0 4f59 b9f2 9135e42f991c" { "name" "ecdb280e 0ff0 4f59 b9f2 9135e42f991c", "type" "tree", "object data" { "num" \[ { "val" 332 0, "name" "classification id" } ] } }, "b088dcbf c7fe 46a5 a71d 88ef49bdc107" { "name" "b088dcbf c7fe 46a5 a71d 88ef49bdc107", "type" "3dpointcloudsegmentation" } } } } geometry collection as a relation here is an example of 2 curves forming a geometry collection note that geometry collections can be any grouping of geometries of any type { "openlabel" { "frames" { "0" { "objects" { "824e20e4 b3bd 4ba8 8d40 9d32a3985312" { "object data" { "poly2d" \[ { "attributes" {}, "closed" false, "mode" "mode poly2d absolute", "name" "curve 518f2936", "val" \[] } ] } }, "923eb0cf ddf2 4eb3 881c 95846d8d1a3d" { "object data" { "poly2d" \[ { "attributes" {}, "closed" false, "mode" "mode poly2d absolute", "name" "curve e63dc662", "val" \[] } ] } }, "f7b8704f 985c 4369 93a8 5b0c77757ee9" { "object data" {} } } } }, "metadata" { "schema version" "1 0 0" }, "objects" { "824e20e4 b3bd 4ba8 8d40 9d32a3985312" { "name" "824e20e4 b3bd 4ba8 8d40 9d32a3985312", "object data" {}, "type" "dashed" // collection subtype 1 }, "923eb0cf ddf2 4eb3 881c 95846d8d1a3d" { "name" "923eb0cf ddf2 4eb3 881c 95846d8d1a3d", "object data" {}, "type" "solid" // collection subtype 2 }, "f7b8704f 985c 4369 93a8 5b0c77757ee9" { "name" "f7b8704f 985c 4369 93a8 5b0c77757ee9", "object data" {}, "type" "lanemarking" // the collection type } }, "relations" { "0" { "name" "0", // specifying the collection as a relation "rdf objects" \[ { "type" "object", "uid" "824e20e4 b3bd 4ba8 8d40 9d32a3985312" }, { "type" "object", "uid" "923eb0cf ddf2 4eb3 881c 95846d8d1a3d" } ], "rdf subjects" \[ { "type" "object", "uid" "f7b8704f 985c 4369 93a8 5b0c77757ee9" } ], "type" "geometry collection" } } } }