Overview

A scene represents a grouping of sensor data (e.g. camera images, lidar pointclouds) that should be annotated together. Any information necessary to describe the relationship between the sensors and their captured data is also specifed in the scene, be it camera resolution, sensor name, and the frequency at which the data was recorded at, etc.

There are different scene types depending on what kind of sensor(s) are used to represent the contents of the scene. For example, if one wants to create a scene only consisting of image data from camera sensors then one would use the scene type Cameras. Similarly, if one wants to create a scene consisting of both lidar and camera sensors then one would use the scene type LidarsAndCameras. Additionally, scenes can either be single frame or sequence type.

Sequential scenes represent a sequence of frames in time, whereas non-sequential scenes only contain one snapshot of the sensor data. The sequential relationship is expressed via a sequence of Frames, where each Frame contains information related to what kind of sensor data constitues the frame (e.g. which image and/or pointcloud is part of the Frame) as well as a relative timestamp that captures where in time (relative to the other frames) the Frame is located.

Non-sequential scenes only contains a single Frame and do not require any relative timestamp information.

Sequential scene types are identified by the suffix Seq in their type name.

The following scene types are currently supported

Non-sequential scenes have the following structure

Sequential scenes are similarly represented, except that they instead contain a list of Frames

A scene automatically gets a UUID when it is created. This UUID is used as the primary identifier by Kognic and all of our internal systems. Additionally, an external id is required as an identifier when creating the scene in order to make communication around specific scenes easier.

The sensor specification contains information about the camera and/or lidar sensors used in the scene.

The additional fields are optional, and can be used to specify the order of the camera images and a human readable sensor name (e.g. "Front Camera" instead of "FC") when viewed in the Kognic annotation App.

As an example, let's say we have three camera sensors R, F and L positioned on the ego vehicle. Creating a sensor specification would be

sensor_order configures the order of camera images, and sensor_to_pretty_name affects the labels when viewed in the Kognic annotation App.

Any scene consisting of lidar and camera sensors requires a calibration. The calibration specifies the spatial relationship (position and rotation) between the sensors, and the camera intrinsic parameters.

However, scenes without a lidar sensor do not require a calibration.

Calibration is used by the Kognic annotation App to project regions in the pointcloud when a camera image is selected, and, similarly, to project the selected object (e.g. point, cuboid) in the pointcloud onto the images .

When creating a calibration, all sensors must match those present on the scene. If this is not the case the scene will not be created and a validation error will be returned by the Kognic API.

Detailed documentation on how to create calibrations via the API is present in the Calibrations Overview.

Metadata can be added to scenes via the metadata field. It consists of flat key-value pairs, which means that nested data structures are not allowed. Metadata can be used to include additional information about a scene. Metadata cannot be seen by the annotators, but there are some reserved keywords that can alter the behaviour of the Kognic annotation tool. Reserved keywords can be found in the MetaData object in the python client.

The Frame object specifies the binary data to be annotated (.jpg, .png, .las etc) as well as which sensor the data originated from. Note that the Frame object is different for each scene type, even though the overall structure is similar (see details below).

As an example, let's say we want to create a scene consiting of images from three camera sensors R, F and L. The corresponding binary data are in the files img_cam_R.jpg, img_cam_F.jpg and img_cam_F.jpg. This would correspond the scene type Cameras.

Similarly, if we also had an associated lidar pointcloud from the sensor VDL-64 and a corresponding binary file scan_vdl_64.las, we would instead use the scene type LidarsAndCameras. Note that the Frame class shall be imported under the corresponding scene type.

Sequential scene takes a list of Frame objects instead of a single Frame. In addition, the Frame object associated with sequential scenes have three additional parameters: frame_id, relative_timestamp and metadata.

The sequential relationship is expressed via the order of the list of Frame.

To express how much time has passed between the different frames, one can use the relative_timestamp parameter for each Frame. The relative timestamp is expressed in milliseconds and describes the relative time between the Frame and the start of the scene.

For example, let's say that the sensor data is collected and aggregated at 2Hz.

The frame_id is a string that uniquely identifies each frame in the list of frames.

A common use case is to use uuids for each frame_id, or a combination of external_id and frame_index. For example, if the external_id of the scene is shanghai_20200101 then the frame_id could be encoded as shanghai_20200101:0 for the first frame, shanghai_20200101:1 for the second frame and so on.

It's also possible to provide metadata on a frame level for sequential frames. It consists of flat key-value pairs and is not exposed to annotators during the production of annotations.

As an example, let's say we want to create a scene of type CamerasSequence consisting of 2 frames, each with camera images from two sensors R and L.

Every file containing sensor data is represented as a Resource, with Image and PointCloud being the concrete subclasses.

Resources ultimately describe how to obtain some binary or textual sensor data, which can be done in different ways:

Resources must always be given a filename. For alternative 1 this must point to the local file to upload. For alternatives 2 & 3 the value of filename parameter is treated as an identifier; it is used to name the uploaded file but does not have to correspond to your filesystem.

Resources always have a sensor_name which identifies the sensor they were captured from. In sequential scenes, each Frame will have a Resource for each sensor.



For alternatives 2 & 3 listed above, a FileData object is attached to the Resource (Image or PointCloud) to specify the source of data. A FileData is created with either data: UploadableData or a callback: Callable[[str], UploadableData], as well as a format which identifies the type of data contained in the bytes. Examples of this are shown below. UploadableData is our type alias for supported sources of raw data: bytes, BinaryIO, IOBase as well as generators and async generators of bytes.

Previous API client releases advertised support for ingesting files from external URIs, such as gs://bucket/path/file. Please contact Kognic if you believe you require this functionality going forward.

Set filename to the path of the local file and do not provide data via other means (directly or callback). The content is uploaded using a content type inferred from the filename suffix.

In addition to filename, provide a FileData object via the file_data attribute, which in turn has an UploadableData as its own data attribute. This example uses raw bytes:

In addition to filename, provide a FileData object via the file_data attribute, with a callback function that produces an UploadableData, e.g.

The callback function (get_png) is a unary function with the following signature.

The callback function is invoked with the Resource.filename as its argument when it is time to upload that single file.

If the callback requires extra arguments then we recommend creating a closure over the additional arguments like this:

Using asynchronous callbacks can be useful to speed up data uploads, especially when the data is not available locally. In the same way as for synchronous callbacks, the callback function is invoked with the Resource.filename as its argument when it is time to upload that single file. Asynchronous callbacks can be used in the following way:

In addition to filename, provide a FileData object via the file_data attribute, which in turn has a bytes generator or async generator as its callback attribute. This example uses an async generator to very slowly stream a local file in small chunks.

Inertial Measurement Unit (IMU) data may be provided for scenes containing LIDAR pointclouds. This can be used to perform motion compensation in multi-lidar setups, and by default if any IMU data is provided this will be done. Motion compensation may be disabled via a scene feature flag, for cases where motion compensation has already been performed prior to upload.

Refer to Motion Compensation for Multi-Lidar Setups.

Control over optional parts of the scene creation process is possible via FeatureFlags that are passed when invoking the create operation on the scene. Refer to the feature flags documentation for details.