Action assist: 3D boxes

6 min

Workflow

Find an object in the point cloud

Does it have a white/peach prediction box? Select it.

Do you get a go to best frame suggestion? Press it. Then select the desired class.

Did you not get a best frame suggestion? Select the desired class.

No prediction box? Draw the object manually with center/side mode.

Check if the object is stationary and set the property. Enable aggregation with alt+a and scrub. - The object is smeared out and appears to leave the location -> Not stationary. - The object points are not smeared out, except for possible drift -> Is stationary.

After you have created the object you will either annotate a stationary object or a dynamic object. And they will have different workflows.

Suggested Actions

Go to next frame - shift+space Jumps to and zooms into the next frame that needs a keyframe, framing the camera around the predicted box. Marks the previous frame as visited.

Accept prediction - R Accepts the predicted box in the current frame as a keyframe.

Go to best predicted frame Offered when you've just selected an object that has no annotation yet, or when the existing keyframe isn't on the best frame for the object. Jumps to the frame where the prediction is clearest (best point coverage) so you can start there.

Move keyframe Offered for stationary objects right after the first keyframe. Replaces the first keyframe with one placed on the best predicted frame, so the rest of the track inherits a cleaner anchor.

The stationary toggle If the task is not configured to include the 'is stationary' property you will get a suggestion to set the object as stationary in the action assist window.

Auto-accept

Toggle Auto-accept in the Action Assist panel to accept predictions as you navigate, so a single shift+space accepts the current suggestion and moves to the next frame in one step. Useful when the predictions look consistently good and you know many keyframes are required. Toggle location is subject to change.

You do not have to select predictions. As you annotate the object any prediction overlapping the current trajectory will automatically be picked up by the system. You only have to follow the highlight box.

The assist walks forward from your last keyframe until the end of the sequence, then backward from your first keyframe to the start — stopping automatically at frames marked first/last for the object.

Visited frames are remembered. Once you step past a frame, the assist won't loop back to it for the same object. Switching objects resets the visited list for the new object.

A single keyframe is usually enough to get a full track. The pipeline extrapolates the velocity from your keyframe(s) and matches to overlapping prelabel predictions to fill the rest of the sequence.

Heading alignment only kicks in when the object is moving fast enough and isn't turning sharply. Slow-moving objects (parking maneuvers, crawling traffic) keep the heading from the prediction or your last keyframe — verify the heading on these manually. For objects that should be treated as fully stationary, use the Stationary toggle instead.

Predictions already used by other objects are skipped. Predictions that another annotation has consumed, or that you've explicitly dismissed via the Copilot ignore list, will not be offered as matches.

A suggestion only appears when there are predictions to extrapolate to, or when the trajectory clearly continues into a frame. If the object exits the scene, becomes occluded for a long stretch, or has no nearby prelabels, no suggestion is generated for that frame — mark the last visible frame as the object's final frame and the assist will stop walking forward.

Ground alignment uses the per-frame ground mesh under the cuboid. In areas where the lidar is sparse and the mesh is unreliable, the pitch/roll correction is skipped and the prediction keeps its original orientation.

For stationary objects, the relative-keyframe predictor only uses anchors within 50 m that are themselves marked stationary and have keyframes covering both the reference frame and the prediction frame. Adding stationary annotations to nearby objects (signs, parked cars) materially improves the assist for everything else stationary in the scene.

3D boxes

Action assist: 2D / 3D fusion

Copilot

Side mode

Center mode

Filter cuboids below point threshold

Point cloud aggregation