Action assist: 2D / 3D fusion

in 2d/3d fusion tasks where you connect 3d boxes and 2d boxes, action assist provides recommendations in the 2d view it is most efficient to annotate an object in the 3d view first this provides a perfect prior for where the object is in all cameras at once sequence first you want to annotate the object fully in 3d using action assist 3d boxes docid\ sbxa8ffxzbll 5fwq7qnl once finished, switch over to the 2d view here you will be guided through the currently selected camera to each frame where the 3d object projects into the camera image, marked with the blue "attention frame" markers in the videobar in order to be productive you must avoid the create connected 2d box docid 8oxmj9rnuv9li 8vh1bdx feature, the projection is always too large making you have to adjust every single 2d box action assist learns how to correct for calibration errors based on the annotated keyframes it will also make action assist stop giving you suggestions and predictions workflow annotate the object in 3d as usual ( action assist 3d boxes docid\ sbxa8ffxzbll 5fwq7qnl ) switch to the camera view press shift+space to jump to the first frame that needs a box, then r to accept the suggestion — or turn on auto accept and just press shift+space through the sequence adjust any boxes that don't look right; subsequent suggestions in nearby frames will follow your adjustment repeat step 3 and 4 until the video bar shows no more highlighted frames for the object verify that the start and end frames are marked correctly how it works select an object that already has at least one 3d cuboid annotation open a camera view for every frame where the object's cuboid appears, the system projects a 2d bounding box into that image if there is an overlapping prediction of similar size, that prediction is used instead of the raw projection — giving you a tighter, more accurate box suggested actions the action assist panel shows the next recommended step for the selected object go to next frame shift+space jumps to and zooms into the next frame that needs a box marks the previous frame as visited accept prediction r accepts the suggested box in the current frame as a keyframe auto accept toggle auto accept in the action assist panel to accept predictions as you navigate, so a single shift+space accepts the current suggestion and moves to the next frame in one step useful when the suggestions look consistently good and you only need to step through to confirm toggle location is subject to change things to know the assist walks forward from your last keyframe until the end of the sequence, then backward from your first keyframe to the start — stopping automatically at frames marked first/last for the object visited frames are remembered once you step past a frame, the assist won't loop back to it for the same object switching objects resets the visited list for the new object larger objects (cars, trucks, buses, etc ) use a tighter mesh based projection rather than the raw cuboid corners, so the suggested box hugs the vehicle more closely if you nudge a box, nearby suggestions follow when you edit a 2d box away from where the cuboid projects to, the assist remembers the angular offset and applies it to suggestions in nearby frames (within 3 seconds) — useful when the camera/lidar calibration drifts persistently in one direction the compensation only kicks in away from the image edges where the geometry is reliable predictions already used by other objects are skipped predictions that another annotation has consumed, or that you've explicitly dismissed via the copilot ignore list, will not be offered as matches a suggestion only appears when the object has a 3d cuboid in that frame if the cuboid is missing in a particular frame, no 2d suggestion is generated for that frame single frame the single frame action assist for 2d / 3d fusion will guide you through the optimal workflow we will be annotating the 3d box first, then draw the object in 2d in each source that the 3d box is visible in next prediction in 3d you will be guided to the next prelabel in the 3d scene if it's a false positive you can simple press 'backspace' to ignore it and press shift+space to go to the next one select class selecting a class with a prelabel selected will create an object at it's location if there is no prelabel for a given object draw it manually go to 2d while you have the context for the current object we encourage you to finish annotating 2d before drawing the next object in 3d draw connected 2d box you will be zoomed to the location of the projection of the 3d box, and the drawing tool for 2d bounding boxes docid\ tfjsznromwyazm4 smr5c will activate if the object is fully occluded press shift+space again to skip this source this step will be repeated for each camera source the object is visible in go back to 3d this action will deselect the current object and go back to 3d going back to step 1 now putting it all together we get the workflow illustrated below related content 3d boxes docid\ jw2jfj5i04upvcc7hv3ek 2d bounding boxes docid\ tfjsznromwyazm4 smr5c action assist 3d boxes docid\ sbxa8ffxzbll 5fwq7qnl create connected 2d box docid 8oxmj9rnuv9li 8vh1bdx