Depth camera 3D pose estimation using 3D CAD models

• US 10,552,981 B2
• Filed: 01/16/2018
• Issued: 02/04/2020
• Est. Priority Date: 01/16/2017
• Status: Active Grant

• Current Assignee: Shapetrace Inc.
• Original Assignee: Shapetrace Inc.
• Inventors: Li-Chee-Ming, Julien, Garad, Adil, Yap, Ernest, Ranjit, Sonal
• Primary Examiner(s): Hon, Ming Y
• Application Number:
  US15/872,313
• Publication Number:
  US 20190221000A1
• Time in Patent Office: 749 Days
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/75 involving models

Abstract

Systems and methods for indoor localization in large-scale scenes, such as indoor environments are described. Systems and related methods for estimating the 3D camera pose of a depth camera by automatically aligning 3D depth images of a scene to a 3D CAD model of the scene are described.

Claims

1. A method of real-time depth camera pose estimation comprising:

at a processor,

receiving a sequence of depth map frames from a moving mobile depth camera, each depth map frame comprising a plurality of image elements, each image element being associated with a depth value related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;

tracking a 3D position and orientation of the mobile depth camera using the depth map frames and a 3D CAD model of the environment, the 3D position and orientation defining a pose of the mobile depth camera, the tracking involving storing the 3D position and orientation of the mobile depth camera in a storage device;

computing, using an initial camera pose estimate, pairs of corresponding corner features between a current depth map frame and the 3D CAD model;

updating the initial camera pose estimate by optimizing an error metric applied to the computed corresponding corner feature pairs;

outputting the updated camera pose estimate; and

wherein computing pairs of corresponding corner features using the initial camera pose estimate comprises;

receiving the initial camera pose estimate, a current depth map, and 3D CAD model corners;

identifying model corners predicted to be in a field of view of the mobile depth camera;

projecting the current depth map onto the 3D CAD model using the initial camera pose estimate to generate a projected depth map;

for each identified model corner,

searching a surrounding area for corresponding corner candidates in the projected death map;

selecting candidate corresponding corners according to a distance metric;

generating four point corner features from the model and the depth map; and

outputting the four point corner features.

2. The method of claim 1 wherein computing the corresponding corner features between the 3D CAD model and current depth map comprises:

using an initial camera pose to project the current depth map onto the 3D CAD model, and identify corners in the 3D CAD model that may be in the camera'"'"'s field of view, wherein a search is performed in each corner'"'"'s surrounding area for the closest corner in the projected depth map.

3. The method of claim 2 wherein extracting at least one of the corner features from the 3D CAD model in an offline phase comprises:

checking combinations of three planes in the 3D CAD model for a point of concurrency.

4. The method of claim 2 wherein at least one of the corner features from a depth frame is extracted by, upon a corner in the 3D CAD model appearing in the camera'"'"'s field of view, performing a search in the model corner'"'"'s surrounding area for a corresponding corner in the projected depth map, extracting planes from the portion of the depth map within the search area, discarding the planes that are not parallel, to within a threshold, to any of the planes of the model'"'"'s corner feature.

5. The method of claim 4 wherein the combinations of three planes is checked for a point of congruency and the method further comprises selecting the three planes whose point of intersection is closest to the model'"'"'s corner.

6. The method of claim 2 wherein at least one of the corner features is generated by:

extracting a group of four points extracted from the intersection of three planes, wherein the first point, defined as an origin of the corner feature, is the point of intersection of all three planes.

7. The method of claim 6 wherein the three lines formed by the three intersecting planes form the basis of the corner feature and the three other points of the corner feature are located one unit length from the origin, each along a separate basis vector.

8. The method of claim 1 further comprising:

computing the initial estimate of camera pose;

calculating an optimal estimate of the camera pose by minimizing or optimizing the error metric applied to the computed corresponding corner features and determining that convergence is reached; and

outputting the optimal estimate of camera pose.

9. A system of real-time depth camera pose estimation comprising a persistent data store storing instructions executable by a processor to:

receive a sequence of depth map frames from a moving mobile depth camera, each depth map frame comprising a plurality of image elements, each image element having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;

track a 3D position and orientation of the mobile depth camera using the depth map frames and a 3D CAD model of the environment, the 3D position and orientation defining a pose of the mobile depth camera, the tracking involving storing the 3D position and orientation of the mobile depth camera in the persistent data store;

compute, using an initial camera pose estimate, pairs of corresponding corner features between a current depth map frame and the 3D CAD model; and

update the estimate of the camera pose by optimizing an error metric applied to the computed corresponding corner feature points;

store one or more depth map frames and the estimate of the camera pose in the persistent data store;

output the estimate of the camera pose; and

wherein to compute pairs of corresponding corner features using the initial camera pose estimate, the system instructs the processor to;

receive the initial camera pose estimate, a current depth map, and 3D CAD model corners;

identify model corners predicted to be in a field of view of the mobile depth camera;

project the current depth mad onto the 3D CAD model using the initial camera pose estimate to generate a projected depth map;

for each identified model corner,

search a surrounding area for corresponding corner candidates in the projected depth select candidate corresponding corners according to a distance metric;

generate four point corner features from the model and the depth map; and

output the four point corner features.

10. The system of claim 9, wherein computing the corresponding corner features between the 3D CAD model and current depth map comprises:

using an initial camera pose to project the current depth map onto the 3D CAD model, and identify corners in the 3D CAD model that may be in the camera'"'"'s field of view, wherein a search is performed in each corner'"'"'s surrounding area for the closest corner in the projected depth map.

11. The system of claim 10 wherein extracting at least one of the corner features from the 3D CAD model in an offline phase comprises:

checking combinations of three planes in the 3D CAD model for a point of concurrency.

12. The system of claim 10 wherein at least one of the corner features from a depth frame is extracted by, upon a corner in the 3D CAD model appearing in the camera'"'"'s field of view, performing a search in the model corner'"'"'s surrounding area for a corresponding corner in the projected depth map, extracting planes from the portion of the depth map within the search area, discarding the planes that are not parallel, to within a threshold, to any of the planes of the model'"'"'s corner feature.

13. The system of claim 12 wherein the combinations of three planes is checked for a point of congruency and the instructions, when executed by the processor, configure the processor to select the three planes whose point of intersection is closest to the model'"'"'s corner.

14. The system of claim 10 wherein at least one of the corner features is generated by:

extracting a group of four points extracted from the intersection of three planes, wherein the first point, called the origin of the corner feature, is the point of intersection of all three planes.

15. The system of claim 14 wherein the three lines formed by the three intersecting planes form the basis of the feature and the three other points of the corner feature are located one unit length from the origin, each along a separate basis vector.

16. The system of claim 9 further comprising a persistent data store storing instructions executable by a processor to:

compute the initial estimate of camera pose;

calculate an optimal estimate of the camera pose by minimizing or optimizing the error metric applied to the computed corresponding corner features and determining that convergence is reached; and

output the optimal estimate of camera pose.

17. A non-transitory computer-readable storage medium comprising computer-executable instructions for causing a processor to compute real-time depth camera pose estimations by:

storing a 3D position and orientation of a mobile depth camera using death map frames and a 3D CAD model of the environment, the 3D position and orientation defining a pose of the mobile depth camera;

forming an initial estimate of camera pose using depth map frames captured by fall the moving mobile depth camera;

computing pairs of corresponding corner features using the initial estimate;

calculating an optimal estimate of the camera pose by minimizing an error metric applied to the computed corresponding corner features;

determining that convergence is reached;

outputting the optimal estimate of camera pose; and

wherein the computing of the pairs of corresponding corner features using the initial camera pose estimate comprises;

receiving the initial camera pose estimate, a current depth map, and 3D CAD model corners;

identifying model corners predicted to be in a field of view of the mobile death camera;

projecting the current death map onto the 3D CAD model using the initial camera pose estimate to generate a projected death map;

for each identified model corner,

searching a surrounding area for corresponding corner candidates in the projected death map;

selecting candidate corresponding corners according to a distance metric;

generating four point corner features from the model and the death map; and

outputting the four point corner features.

18. The non-transitory computer-readable storage medium of claim 17 wherein each depth map frame comprises a plurality of image elements, each image element being associated with a depth value related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera.