Method and device for high-resolution three-dimensional imaging which obtains camera pose using defocusing

• US 20090295908A1
• Filed: 05/21/2009
• Published: 12/03/2009
• Est. Priority Date: 01/22/2008
• Status: Active Grant

• Current Assignee: California Institute of Technology
• Original Assignee: California Institute of Technology
• Inventors: Lu, Jian, Harvard, Alexei, Gharib, Morteza, Hsieh, Scott, Pereira, Francisco
• Granted Patent: US 8,514,268 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G01B 11/002 for measuring two or more coordinates
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/236 using varifocal lenses or mirrors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects

Abstract

A method and device for high-resolution three-dimensional (3-D) imaging which obtains camera pose using defocusing is disclosed. The device comprises a lens obstructed by a mask having two sets of apertures. The first set of apertures produces a plurality of defocused images of the object, which are used to obtain camera pose. The second set of optical filters produces a plurality of defocused images of a projected pattern of markers on the object. The images produced by the second set of apertures are differentiable from the images used to determine pose, and are used to construct a detailed 3-D image of the object. Using the known change in camera pose between captured images, the 3-D images produced can be overlaid to produce a high-resolution 3-D image of the object.

Claims

1. A method for determining a change in pose of a moving sensor using a defocusing technique comprising acts of:

capturing, at an initial time, an initial plurality of defocused images of an object substantially simultaneously with a sensor, from an initial sensor pose;

extracting three-dimensional (3-D) locations of three or more object features from the relative locations of the object features in the initial plurality of defocused images on the sensor;

capturing, at a subsequent time, a subsequent plurality of defocused images of the object substantially simultaneously, from a subsequent sensor pose;

extracting 3-D locations of three or more object features from the relative locations of the object features in the subsequent plurality of defocused images on the sensor;

matching the object features from the initial plurality of defocused images with corresponding object features from the subsequent plurality of defocused images using a technique selected from the group consisting of;

a feature matching algorithm; and

an error minimization method; and

calculating a change in pose of the sensor between the initial and subsequent times using the 3-D locations extracted from the initial and subsequent pluralities of defocused images,

whereby the change in pose of the moving sensor is determined.

2. The method of claim 1, further comprising an act of constructing a high-resolution 3-D image of the object by:

projecting a pattern of markers on the object;

capturing, at the initial time, and from the initial sensor pose, an initial plurality of defocused images of the projected pattern of markers with the sensor, the defocused images being differentiable from the initial pluralities of defocused images of the object used for determining sensor pose;

constructing a 3-D image of the object based on relative positions of the initial plurality of defocused images of the projected pattern of markers on the sensor;

capturing, at the subsequent time, and from the subsequent sensor pose, a subsequent plurality of defocused images of the projected pattern of markers with the sensor, the defocused images being differentiable from the subsequent pluralities of defocused images of the object used for determining sensor pose;

constructing a 3-D image of the object based on relative positions of the subsequent plurality of defocused images of the projected pattern of markers on the sensor; and

overlaying the 3-D images constructed from the initial and subsequent pluralities of defocused images of the projected pattern of markers using the known change in sensor pose between the initial and subsequent times to produce a high-resolution 3-D image of the object.

3. The method of claim 2, further comprising an act of resolving the detailed 3-D image of the object to a desired resolution by:

capturing, at a second subsequent time, a second subsequent plurality of defocused images of the object substantially simultaneously, from a second subsequent sensor pose;

extracting 3-D locations of three or more object features from the relative locations of the object features in the second subsequent plurality of defocused images on the sensor;

matching the object features from the subsequent plurality of defocused images with corresponding object features from the second subsequent plurality of defocused images using a technique selected from the group consisting of;

a feature matching algorithm; and

an error minimization method; and

calculating a change in pose of the sensor between the subsequent and second subsequent times using the 3-D locations extracted from the subsequent and second subsequent pluralities of defocused images;

projecting a pattern of markers on the object;

capturing, at the second subsequent time, and from the second subsequent sensor pose, a second subsequent plurality of defocused images of the projected pattern of markers with the sensor, the defocused images being differentiable from the second subsequent pluralities of defocused images of the object used for determining sensor pose;

constructing a 3-D image of the object based on relative positions of the second subsequent plurality of defocused images of the projected pattern of markers on the sensor;

overlaying the 3-D images constructed from the initial, subsequent, and second subsequent pluralities of defocused images of the projected pattern of markers using the known change in sensor pose between the initial and second subsequent times to produce a high-resolution 3-D image of the object; and

repeating these acts at further subsequent times and at further subsequent camera poses until a desired resolution is reached.

4. The method of claim 3, further comprising an act of calculating an absolute pose of the sensor with respect to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor; and

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.

5. The method of claim 1, further comprising an act of calculating an absolute pose of the sensor with respect to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor; and

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.

6. An imaging device for producing a high-resolution three-dimensional (3-D) image of an object, comprising:

a lens obstructed by a mask having at least one set of off-axis apertures for producing a plurality of defocused images of an object substantially simultaneously;

a sensor configured to capture the plurality of defocused images produced; and

;

a data processing system having one or more processors configured to;

capture, at an initial time, an initial plurality of defocused images of an object substantially simultaneously with a sensor, from an initial sensor pose;

extract three-dimensional (3-D) locations of one or more object features from the relative locations of the object features in the initial plurality of defocused images on the sensor;

capture, at a subsequent time, a subsequent plurality of defocused images of the object substantially simultaneously, from a subsequent sensor pose;

match the object features from the initial plurality of defocused images with corresponding object features from the subsequent plurality of defocused images using a technique selected from the group consisting of;

a feature matching algorithm; and

an error minimization method; and

calculate a change in pose of the sensor between the initial and subsequent times using the 3-D locations extracted from the initial plurality of defocused images,

whereby the change in pose of the moving sensor is determined.

7. The imaging device of claim 6, further comprising a projector for projecting a pattern of markers on the surface of the object, the projected pattern of markers being differentiable from the object features on the object, and wherein the data processing system is further configured to:

project a pattern of markers on the object;

capture a plurality of defocused images of the projected pattern of markers with the sensor, the defocused images being differentiable from the initial and subsequent pluralities of defocused images of the object features used for determining sensor pose; and

construct a 3-D image of a surface of the object based on relative positions of the plurality of defocused images of the projected pattern of markers on the sensor.

8. The imaging device of claim 7, wherein the data processing system is further configured to calculate an absolute pose of the sensor relative to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor;

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.

9. The imaging device of claim 6, wherein the data processing system is further configured to calculate an absolute pose of the sensor relative to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor;

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.

10. An imaging device for producing a high-resolution three-dimensional (3-D) image of an object, comprising;

a lens obstructed by a mask having a first set and a second set of off-axis apertures where;

the first set of off-axis apertures comprises a plurality of apertures fitted with filter separators for capturing a plurality of defocused images of an object substantially simultaneously; and

the second set of off-axis apertures comprises a plurality of apertures fitted with filter separators differentiable from those of the first set of off-axis apertures for capturing a plurality of defocused images,

a projector for projecting a pattern of markers on the surface of the object, the projected pattern of markers being of a wavelength corresponding to the filter separators of the second set of off-axis apertures; and

a sensor configured to capture the defocused images produced,

whereby the images captured on the sensor through the first set of off-axis apertures can be used to determine camera pose, and the images captured on the sensor through the second set of off-axis apertures can be used to construct a high-resolution three-dimensional image of the object.

11. The device of claim 10, wherein the device further comprises a data processing system, the data processing system having one or more processors configured to:

capture, at an initial time, an initial plurality of defocused images of an object substantially simultaneously with a sensor, from an initial sensor pose;

extract three-dimensional (3-D) locations of three or more object features from the relative locations of the object features in the initial plurality of defocused images on the sensor;

capture, at a subsequent time, a subsequent plurality of defocused images of the object substantially simultaneously, from a subsequent sensor pose;

extract three-dimensional (3-D) locations of three or more object features from the relative locations of the object features in the subsequent plurality of defocused images on the sensor;

match the object features from the initial plurality of defocused images with corresponding object features from the subsequent plurality of defocused images using a technique selected from the group consisting of;

a feature matching algorithm; and

an error minimization method; and

calculate a change in pose of the sensor between the initial and subsequent times using the 3-D locations extracted from the initial and subsequent pluralities of defocused images,

whereby the change in pose of the moving sensor is determined.

12. The device of claim 11, wherein the data processing system is further configured to:

project a pattern of markers on the object;

capture a plurality of defocused images of the projected pattern of markers with the sensor, the defocused images being differentiable from the initial and subsequent pluralities of defocused images of the object used for determining sensor pose; and

construct a 3-D image of a surface of the object based on relative positions of the plurality of defocused images of the projected pattern of markers on the sensor.

13. The device of claim 12, wherein the data processing system is further configured to calculate an absolute pose of the sensor relative to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor;

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.

14. The device of claim 13, wherein each of the plurality of apertures comprising the first set of off-axis apertures is fitted with a different type of filter separator, such that the defocused images produced through said apertures are differentiable from one another.

15. The device of claim 11, wherein the data processing system is further configured to calculate an absolute pose of the sensor relative to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor;

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.

16. The device of claim 10, wherein each of the plurality of apertures comprising the first set of off-axis apertures is fitted with a different type of filter separator, such that the defocused images produced through said apertures are differentiable from one another.

17. A computer program product for determining a change in pose of a moving sensor using a defocusing technique, comprising computer-readable instruction means stored on a computer-readable medium that are executable by a computer for causing the computer to:

capture, at an initial time, an initial plurality of defocused images of an object substantially simultaneously with a sensor, from an initial sensor pose;

extract three-dimensional (3-D) locations of three or more object features from the relative locations of the object features in the initial plurality of defocused images on the sensor;

capture, at a subsequent time, a subsequent plurality of defocused images of the object substantially simultaneously, from a subsequent sensor pose;

extract three-dimensional (3-D) locations of three or more object features from the relative locations of the object features in the subsequent plurality of defocused images on the sensor;

match the object features from the initial plurality of defocused images with corresponding object features from the subsequent plurality of defocused images using a technique selected from the group consisting of;

a feature matching algorithm; and

an error minimization method; and

calculate a change in pose of the sensor between the initial and subsequent times using the 3-D locations extracted from the initial and subsequent pluralities of defocused images,

whereby the change in pose of the moving sensor is determined.

18. The computer program product of claim 17, further comprising computer-readable instructions for causing the computer to:

project a pattern of markers on the object;

capture a plurality of defocused images of the projected pattern of markers with the sensor, the defocused images being differentiable from the initial and subsequent pluralities of defocused images of the object used for determining sensor pose; and

construct a 3-D image of a surface of the object based on relative positions of the plurality of defocused images of the projected pattern of markers on the sensor.

19. The computer program product of claim 18, further comprising computer-readable instructions for causing the computer to calculate an absolute pose of the sensor relative to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor;

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.

20. The computer program product of claim 17, further comprising computer-readable instructions for causing the computer to calculate an absolute pose of the sensor relative to an environment by:

capturing a plurality of defocused images of three or more fixed features of known position in the environment with the sensor;

extracting 3-D locations of the three or more fixed features in the environment from the relative locations of the fixed features in the plurality of defocused images on the sensor; and

calculating the absolute pose of the sensor using the 3-D locations extracted from the plurality of defocused images.