Overview

different types of scenes a scene represents a grouping of sensor data (e g camera images, lidar pointclouds) that should be annotated together any information necessary to describe the relationship between the sensors and their captured data is also specifed in the scene, be it camera resolution, sensor name, and the frequency at which the data was recorded at, etc there are different scene types depending on what kind of sensor(s) are used to represent the contents of the scene for example, if one wants to create a scene only consisting of image data from camera sensors then one would use the scene type cameras similarly, if one wants to create a scene consisting of both lidar and camera sensors then one would use the scene type lidarsandcameras additionally, scenes can either be single frame or sequence type sequential vs non sequential sequential scenes represent a sequence of frames in time, whereas non sequential scenes only contain one snapshot of the sensor data the sequential relationship is expressed via a sequence of frames , where each frame contains information related to what kind of sensor data constitues the frame (e g which image and/or pointcloud is part of the frame) as well as a relative timestamp that captures where in time (relative to the other frames) the frame is located non sequential scenes only contains a single frame and do not require any relative timestamp information sequential scene types are identified by the suffix seq in their type name the following scene types are currently supported cameras lidarsandcameras camerasseq lidarsandcamerasseq aggregatedlidarsandcamerasseq scene fields non sequential scenes have the following structure class scene(basemodel) external id str frame frame sensor specification sensorspecification calibration id optional\[str] # required if using lidar sensors metadata mapping\[str, union\[int, float, str, bool]] = field(default factory=dict) sequential scenes are similarly represented, except that they instead contain a list of frames class sceneseq(basemodel) external id str frames list\[frame] sensor specification sensorspecification calibration id optional\[str] # required if using lidar sensors metadata mapping\[str, union\[int, float, str, bool]] = field(default factory=dict) external id a scene automatically gets a uuid when it is created this uuid is used as the primary identifier by kognic and all of our internal systems additionally, an external id is required as an identifier when creating the scene in order to make communication around specific scenes easier sensor specification the sensor specification contains information about the camera and/or lidar sensors used in the scene the additional fields are optional, and can be used to specify the order of the camera images and a human readable sensor name (e g "front camera" instead of "fc") when viewed in the kognic annotation app as an example, let's say we have three camera sensors r , f and l positioned on the ego vehicle creating a sensor specification would be from kognic io model import sensorspecification sensor spec = sensorspecification( sensor to pretty name={ "r" "right camera", "f" "front camera", "l" "left camera" }, sensor order=\["l", "f", "r"] ) sensor order configures the order of camera images, and sensor to pretty name affects the labels when viewed in the kognic annotation app calibration any scene consisting of lidar and camera sensors requires a calibration the calibration specifies the spatial relationship (position and rotation) between the sensors, and the camera intrinsic parameters however, scenes without a lidar sensor do not require a calibration calibration is used by the kognic annotation app to project regions in the pointcloud when a camera image is selected, and, similarly, to project the selected object (e g point, cuboid) in the pointcloud onto the images when creating a calibration, all sensors must match those present on the scene if this is not the case the scene will not be created and a validation error will be returned by the kognic api detailed documentation on how to create calibrations via the api is present in the calibrations overview docid 4mfc9atwcxupflne7v1iz metadata metadata can be added to scenes via the metadata field it consists of flat key value pairs, which means that nested data structures are not allowed metadata can be used to include additional information about a scene metadata cannot be seen by the annotators, but there are some reserved keywords that can alter the behaviour of the kognic annotation tool reserved keywords can be found in the metadata object in the python client frame the frame object specifies the binary data to be annotated ( jpg, png, las etc) as well as which sensor the data originated from note that the frame object is different for each scene type, even though the overall structure is similar (see details below) non sequential frame as an example, let's say we want to create a scene consiting of images from three camera sensors r , f and l the corresponding binary data are in the files img cam r jpg , img cam f jpg and img cam f jpg this would correspond the scene type cameras from kognic io model scene resources import image from kognic io model scene cameras import cameras, frame cameras scene = cameras( , frame=frame( images=\[ image("img cam r jpg", sensor name="r"), image("img cam f jpg", sensor name="f"), image("img cam l jpg", sensor name="l"), ] ) ) similarly, if we also had an associated lidar pointcloud from the sensor vdl 64 and a corresponding binary file scan vdl 64 las , we would instead use the scene type lidarsandcameras note that the frame class shall be imported under the corresponding scene type from kognic io model scene resources import image, pointcloud from kognic io model scene lidars and cameras import lidarsandcameras, frame lidars and cameras = lidarsandcameras( , frame=frame( images=\[ image("img cam r jpg", sensor name="r"), image("img cam f jpg", sensor name="f"), image("img cam l jpg", sensor name="l"), ], point clouds=\[ pointcloud("scan vdl 64 las", sensor name="vdl 64") ] ) ) sequential frames sequential scene takes a list of frame objects instead of a single frame in addition, the frame object associated with sequential scenes have three additional parameters frame id , relative timestamp and metadata the sequential relationship is expressed via the order of the list of frame to express how much time has passed between the different frames, one can use the relative timestamp parameter for each frame the relative timestamp is expressed in milliseconds and describes the relative time between the frame and the start of the scene for example, let's say that the sensor data is collected and aggregated at 2hz frame 1 = frame( , relative timestamp=0) frame 2 = frame( , relative timestamp=500) frame 3 = frame( , relative timestamp=1000) frames = \[frame 1, frame 2, frame 3] the frame id is a string that uniquely identifies each frame in the list of frames a common use case is to use uuids for each frame id , or a combination of external id and frame index for example, if the external id of the scene is shanghai 20200101 then the frame id could be encoded as shanghai 20200101 0 for the first frame, shanghai 20200101 1 for the second frame and so on it's also possible to provide metadata on a frame level for sequential frames it consists of flat key value pairs and is not exposed to annotators during the production of annotators as an example, let's say we want to create a scene of type camerassequence consisting of 2 frames, each with camera images from two sensors r and l from kognic io model scene resources import image from kognic io model scene cameras sequence import camerassequence, frame frames = \[ frame( frame id="1", relative timestamp=0, images=\[ image("img l 1 jpg", sensor name='l'), image("img r 1 jpg", sensor name='r') ]), frame( frame id="2", relative timestamp=500, images=\[ image("img l 2 jpg", sensor name='l'), image("img r 2 jpg", sensor name='r') ]) ] cameras sequence = camerassequence(frames=frames, ) image & pointcloud resources every file containing sensor data is represented as a resource , with image and pointcloud being the concrete subclasses class resource(abc, baseserializer) filename str resource id optional\[str] = none sensor name str file data optional\[filedata] = field(default=none, exclude=true) resource s ultimately describe how to obtain some binary or textual sensor data, which can be done in different ways indirectly by refering to a local filename that contains the data directly provide some bytes like object at creation time lazily provide a callback function which can provide the bytes later in the process resource s must always be given a filename for alternative 1 this must point to the local file to upload for alternatives 2 & 3 the value of filename parameter is treated as an identifier; it is used to name the uploaded file but does not have to correspond to your filesystem resource s always have a sensor name which identifies the sensor they were captured from in sequential scenes, each frame will have a resource for each sensor for alternatives 2 & 3 listed above, a filedata object is attached to the resource ( image or pointcloud ) to specify the source of data a filedata is created with either data uploadabledata or a callback callable\[\[str], uploadabledata] , as well as a format which identifies the type of data contained in the bytes examples of this are shown below uploadabledata is our type alias for supported sources of raw data bytes , binaryio , iobase as well as generators and async generators of bytes previous api client releases advertised support for ingesting files from external uris, such as gs\ //bucket/path/file please contact kognic if you believe you require this functionality going forward local file set filename to the path of the local file and do not provide data via other means (directly or callback) the content is uploaded using a content type inferred from the filename suffix image(filename="/path/to/images/img fc png", sensor name="fc") data in memory in addition to filename , provide a filedata object via the file data attribute, which in turn has an uploadabledata as its own data attribute this example uses raw bytes image( filename="fc frame15", sensor name="fc", file data=filedata(data=b'some png bytes', format=filedata format png) ) data from callback in addition to filename , provide a filedata object via the file data attribute, with a callback function that produces an uploadabledata , e g image( filename="fc frame15", sensor name="fc", file data=filedata(callback=get png, format=filedata format png) ) the callback function ( get png ) is a unary function with the following signature def get png(filename str) > uploadabledata pass the callback function is invoked with the resource filename as its argument when it is time to upload that single file if the callback requires extra arguments then we recommend creating a closure over the additional arguments like this def get callback(arg1, arg2, kwargs) def callback(filename) > bytes \# use arg1, arg2, filename and kwargs return callback filedata( callback=get callback("foo", "bar", extra1="baz", extra2="qux"), format=filedata format jpg ) data from asynchronous callback using asynchronous callbacks can be useful to speed up data uploads, especially when the data is not available locally in the same way as for synchronous callbacks, the callback function is invoked with the resource filename as its argument when it is time to upload that single file asynchronous callbacks can be used in the following way async def get png(filename str) > uploadabledata pass image( filename="fc frame15", sensor name="fc", file data=filedata(callback=get png, format=filedata format png) ) data stream in addition to filename , provide a filedata object via the file data attribute, which in turn has a bytes generator or async generator as its callback attribute this example uses an async generator to very slowly stream a local file in small chunks async def slow stream(filename str) > asyncgenerator\[bytes, any] with open(filename, "rb") as f while chunk = f read(1024) asyncio sleep(1) yield chunk image( filename="fc frame15", sensor name="fc", file data=filedata(callback=slow stream, format=filedata format png) ) imu data inertial measurement unit (imu) data may be provided for scenes containing lidar pointclouds this can be used to perform motion compensation in multi lidar setups, and by default if any imu data is provided this will be done motion compensation may be disabled via a scene feature flags docid\ o snljd9uvm4cnf9eabmv , for cases where motion compensation has already been performed prior to upload refer to motion compensation docid\ rhbsetwlntflfo6fmd 1e scene feature flags control over optional parts of the scene creation process is possible via featureflags that are passed when invoking the create operation on the scene refer to scene feature flags docid\ o snljd9uvm4cnf9eabmv for details