3-DIMENSIONAL SCENE ANALYSIS FOR AUGMENTED REALITY OPERATIONS
First Claim
1. A processor-implemented method for 3-Dimensional (3D) scene analysis, the method comprising:
- receiving, by a processor, a plurality of 3D image frames of a scene, each frame comprising a red-green-blue (RGB) image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames;
projecting, by the processor, the depth pixels into points in a global coordinate system based on the camera pose;
accumulating, by the processor, the projected points into a 3D reconstruction of the scene;
detecting, by the processor, objects and associated locations in the scene, for each 3D image frame, based on the camera pose, the 3D reconstruction, the RGB image frame and the depth map frame;
segmenting, by the processor, each of the detected objects in the scene, the segmented objects comprising the points of the 3D reconstruction corresponding to contours of the associated detected object; and
registering, by the processor, the segmented objects to a 3D model of the associated detected object to determine an alignment of the detected object in the scene.
1 Assignment
0 Petitions
Accused Products
Abstract
Techniques are provided for 3D analysis of a scene including detection, segmentation and registration of objects within the scene. The analysis results may be used to implement augmented reality operations including removal and insertion of objects and the generation of blueprints. An example method may include receiving 3D image frames of the scene, each frame associated with a pose of a depth camera, and creating a 3D reconstruction of the scene based on depth pixels that are projected and accumulated into a global coordinate system. The method may also include detecting objects, and associated locations within the scene, based on the 3D reconstruction, the camera pose and the image frames. The method may further include segmenting the detected objects into points of the 3D reconstruction corresponding to contours of the object and registering the segmented objects to 3D models of the objects to determine their alignment.
-
Citations
25 Claims
-
1. A processor-implemented method for 3-Dimensional (3D) scene analysis, the method comprising:
-
receiving, by a processor, a plurality of 3D image frames of a scene, each frame comprising a red-green-blue (RGB) image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames; projecting, by the processor, the depth pixels into points in a global coordinate system based on the camera pose; accumulating, by the processor, the projected points into a 3D reconstruction of the scene; detecting, by the processor, objects and associated locations in the scene, for each 3D image frame, based on the camera pose, the 3D reconstruction, the RGB image frame and the depth map frame; segmenting, by the processor, each of the detected objects in the scene, the segmented objects comprising the points of the 3D reconstruction corresponding to contours of the associated detected object; and registering, by the processor, the segmented objects to a 3D model of the associated detected object to determine an alignment of the detected object in the scene. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A system for 3-Dimensional (3D) scene analysis, the system comprising:
-
a 3D reconstruction circuit to receive a plurality of 3D image frames of a scene, each frame comprising a red-green-blue (RGB) image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames, the 3D reconstruction circuit further to project the depth pixels into points in a global coordinate system based on the camera pose and accumulate the projected points into a 3D reconstruction of the scene; an object detection circuit to detect objects and associated locations in the scene, for each 3D image frame, based on the camera pose, the 3D reconstruction, the RGB image frame and the depth map frame; a 3D segmentation circuit to segment each of the detected objects in the scene, the segmented objects comprising the points of the 3D reconstruction corresponding to contours of the associated detected object; and a 3D registration circuit to register the segmented objects to a 3D model of the associated detected object to determine an alignment of the detected object in the scene. - View Dependent Claims (11, 12, 13, 14, 15, 16, 17)
-
-
18. At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for 3-Dimensional (3D) scene analysis, the operations comprising:
-
receiving a plurality of 3D image frames of a scene, each frame comprising a red-green-blue (RGB) image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames; projecting the depth pixels into points in a global coordinate system based on the camera pose; accumulating the projected points into a 3D reconstruction of the scene; detecting objects and associated locations in the scene, for each 3D image frame, based on the camera pose, the 3D reconstruction, the RGB image frame and the depth map frame; segmenting each of the detected objects in the scene, the segmented objects comprising the points of the 3D reconstruction corresponding to contours of the associated detected object; and registering the segmented objects to a 3D model of the associated detected object to determine an alignment of the detected object in the scene. - View Dependent Claims (19, 20, 21, 22, 23, 24, 25)
-
Specification