METHODS FOR AUTOMATIC REGISTRATION OF 3D IMAGE DATA

• US 20150332464A1
• Filed: 05/19/2015
• Published: 11/19/2015
• Est. Priority Date: 05/19/2014
• Status: Abandoned Application

• Current Assignee: Occipital, Inc.
• Original Assignee: Occipital, Inc.
• Inventors: O'Keefe, Patrick, Powers, Jeffrey, Burrus, Nicolas
• Application Number:
  US14/716,640
• Publication Number:
  US 20150332464A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0077 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

A method for automatic registration of 3D image data, captured by a 3D image capture system having an RGB camera and a depth camera, includes capturing 2D image data with the RGB camera at a first pose; capturing depth data with the depth camera at the first pose; performing an initial registration of the RGB camera to the depth camera; capturing 2D image data with the RGB camera at a second pose; capturing depth data at the second pose; and calculating an updated registration of the RGB camera to the depth camera.

Claims

1. A method for automatic registration of 3D image data captured by a 3D image capture system having an RGB camera and a depth camera, the method comprising:

capturing 2D image data with the RGB camera at a first pose;

capturing depth data with the depth camera at the first pose;

performing an initial registration of the RGB camera to the depth camera;

capturing 2D image data with the RGB camera at a second pose;

capturing depth data at the second pose; and

calculating an updated registration of the RGB camera to the depth camera.

2. The method of claim 1, wherein calculating the updated registration comprises:

identifying a set of points common to the depth data captured at the first pose and the depth data captured at the second pose;

calculating a color/texture error function for the set of points; and

updating registration data to reduce the color/texture error function.

3. The method of claim 2, wherein the depth camera is a structured light depth camera;

wherein capturing depth data with the depth camera at the first pose comprises capturing depth data using a structured light technique.

4. The method of claim 3, wherein capturing depth data at the second pose comprises capturing depth data with the depth camera using a structured light technique.

5. The method of claim 3, further comprising tracking motion of the 3D image capture system between the first and second poses;

wherein capturing depth data at the second pose comprises capturing depth data using a derived data technique;

wherein using the derived data technique comprises calculating depth data at the second pose from the depth data captured at the first pose and the tracked motion.

6. The method of claim 5, wherein tracking motion comprises tracking motion using an inertial measurement unit.

7. The method of claim 5, wherein tracking motion comprises tracking motion using an iterative closest point tracking technique.

8. The method of claim 2, wherein the depth camera is a stereo depth camera system comprising two infrared cameras;

wherein capturing depth data with the depth camera at the first pose comprises capturing depth data using a stereo technique.

9. The method of claim 2, wherein the depth camera is an assisted stereo depth camera system comprising two infrared cameras and a structured light projector;

wherein capturing depth data with the depth camera at the first pose comprises capturing depth data using an assisted stereo technique.

10. The method of claim 2, wherein capturing 2D image data at the first pose comprises capturing 2D image data at a first time;

wherein capturing 2D image data at the second pose comprises capturing 2D image data after a threshold period of time has elapsed since the first time.

11. The method of claim 2, wherein capturing 2D image data at the first pose comprises capturing 2D image data at a first time;

wherein capturing 2D image data at the second pose comprises capturing 2D image data after;

a threshold period of time has elapsed since the first time, and

motion has been detected by the 3D image capture system after the threshold period of time has elapsed.

12. The method of claim 2, wherein identifying the set of points common to the depth data captured at the first pose and the depth data captured at the second pose comprises identifying the set of points using an iterative closest point algorithm.

13. The method of claim 2, further comprising tracking motion of the 3D image capture system between the first and second poses;

wherein identifying the set of points common to the depth data captured at the first pose and the depth data captured at the second pose comprises identifying the set of points by applying a rigid body transform based on the tracked motion.

14. The method of claim 2, wherein calculating the color/texture error function comprises calculating a mean squared error function for the RGB values of each of the set of points.

15. The method of claim 14, wherein updating registration data to reduce the color/texture error function comprises performing bounded iterative linearization.

16. The method of claim 2, further comprising temporally synchronizing the RGB camera and the depth camera using timing signals.

17. The method of claim 16, wherein temporally synchronizing the RGB camera and the depth camera comprises synchronizing the RGB camera to capture two frames for every frame captured by the depth camera.

18. The method of claim 2, wherein performing the initial registration of the RGB camera to the depth camera comprises performing the initial registration using an image feature detection technique.

19. The method of claim 2, wherein calculating the updated registration comprises calculating the updated registration according to a primary autoregistration technique;

further comprising supplementing automatic registration by performing a secondary autoregistration technique;

wherein the secondary autoregistration technique is distinct from the primary autoregistration technique.

20. The method of claim 19, wherein supplementing automatic registration is performed in response to at least one of environmental conditions, capture system conditions, and performance conditions.

21. The method of claim 2, further comprising supplementing automatic registration by receiving user input corresponding to one or more registration parameters and calculating a refined registration of the RGB camera to the depth camera based on the updated registration and the user input.

22. The method of claim 21, wherein receiving user input comprises displaying a depth map image overlaid on an RGB image, enabling alignment of the depth map image and the RGB image, and measuring transformations performed by a user to align the depth map image and the RGB image.

23. The method of claim 22, wherein enabling alignment of the depth map image and the RGB image comprises enabling translation of the depth map image relative to the RGB image and no other transformation.

24. The method of claim 22, wherein calculating the refined registration comprises calculating the refined registration based on measured transformations performed by the user to align the depth map image and the RGB image.

25. The method of claim 2, further comprising supplementing automatic registration by receiving user input corresponding to one or more registration parameters and calculating an intermediate registration of the RGB camera to the depth camera based on the user input;

wherein calculating the updated registration of the RGB camera to the depth camera comprises calculating the updated registration based on the intermediate registration.

26. The method of claim 25, wherein receiving user input comprises displaying a depth map image overlaid on an RGB image, enabling alignment of the depth map image and the RGB image, and measuring transformations performed by a user to align the depth map image and the RGB image.

27. The method of claim 1, wherein capturing 2D image data with the RGB camera at the first pose comprises capturing an RGB image with the RGB camera, wherein capturing depth data with the depth camera at the first pose comprises capturing an infrared image with the depth camera, wherein calculating the updated registration comprises:

identifying a set of points common to the RGB image and the infrared image;

calculating a transformation between the RGB image and the infrared image; and

calculating the updated registration based on the transformation.

28. The method of claim 27, further comprising calculating the updated registration based on a second transformation, the second transformation describing a relationship between the infrared image and a depth map produced by the depth camera.

29. The method of claim 27, wherein the depth camera is a stereo depth camera system comprising two infrared cameras;

wherein capturing depth data with the depth camera at the first pose further comprises capturing depth data using a stereo technique.

30. The method of claim 27, wherein the depth camera is an assisted stereo depth camera system comprising two infrared cameras and a structured light projector;

wherein capturing depth data with the depth camera at the first pose further comprises capturing depth data using an assisted stereo technique.