System and technique for retrieving depth information about a surface by projecting a composite image of modulated light patterns

• US 7,440,590 B1
• Filed: 05/21/2003
• Issued: 10/21/2008
• Est. Priority Date: 05/21/2002
• Status: Expired due to Fees

• Current Assignee: University Of Kentucky Research Foundation (University of Kentucky)
• Original Assignee: University Of Kentucky Research Foundation (University of Kentucky)
• Inventors: Lau, Daniel L., Guan, Chun, Hassebrook, Laurence G.
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US10/444,033
• Publication Number:
  US 20080279446A1
• Time in Patent Office: 1,980 Days
• Field of Search: 382/106, 382/108, 382/154, 382/100, 352/47, 352/67, 352/41, 359/263, 359/318, 353/34, 720/724, 356/3, 356/3.01, 356/3.09, 345582-588
• US Class Current: 382/108
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

A technique, associated system and program code, for retrieving depth information about at least one surface of an object. Core features include: projecting a composite image comprising a plurality of modulated structured light patterns, at the object; capturing an image reflected from the surface; and recovering pattern information from the reflected image, for each of the modulated structured light patterns. Pattern information is preferably recovered for each modulated structured light pattern used to create the composite, by performing a demodulation of the reflected image. Reconstruction of the surface can be accomplished by using depth information from the recovered patterns to produce a depth map/mapping thereof. Each signal waveform used for the modulation of a respective structured light pattern, is distinct from each of the other signal waveforms used for the modulation of other structured light patterns of a composite image; these signal waveforms may be selected from suitable types in any combination of distinct signal waveforms, provided the waveforms used are uncorrelated with respect to each other. The depth map/mapping to be utilized in a host of applications, for example: displaying a 3-D view of the object; virtual reality user-interaction interface with a computerized device; face—or other animal feature or inanimate object—recognition and comparison techniques for security or identification purposes; and 3-D video teleconferencing/telecollaboration.

Claims

1. A method for retrieving depth information about at least one surface of an object, comprising the steps of:

modulating each of a plurality of structured light patterns using a respective carrier frequency along the orthogonal dimension;

projecting a composite image comprising said plurality of modulated structured light patterns, at the object;

capturing an image reflected from the surface; and

recovering pattern information from said reflected image, for each of said plurality of modulated structured light patterns.

2. The method of claim 1 wherein said step of recovering said pattern information comprises performing a demodulation of said reflected image, said pattern information so recovered from said plurality of structured light patterns comprising the depth information;

and further comprising the step of reconstructing the surface using the depth information.

3. The method of claim 2 wherein:

each said modulated structured light pattern was obtained by modulation with a signal waveform associated with said respective carrier frequency; and

said step of reconstructing comprises producing a depth map of the surface using said recovered pattern information.

4. The method of claim 3 wherein:

each said signal waveform used for said modulation of a respective structured light pattern, is distinct from each other of said signal waveforms used for said modulation of other structured light patterns; and

each said distinct signal waveform is selected from the group consisting of frequency shifted waveforms, phase shifted waveforms, random noise waveforms, and encoded waveforms.

5. The method of claim 2 wherein:

said step of performing a demodulation comprises performing said demodulation along said orthogonal dimension producing a recovered pattern information for each of said plurality of modulated structured light patterns; and

said step of reconstructing comprises producing a depth map of the surface using said recovered pattern information.

6. The method of claim 5 further comprising the step of comparing said depth map with a plurality of stored depth map records of a feature for purposes of identification of the object.

7. The method of claim 5 wherein each said modulated structured light pattern was obtained by modulation with a distinct signal waveform associated with said respective carrier frequency;

and further comprising the step of obtaining orientation data about the object from the depth map.

8. The method of claim 7 wherein said orientation data obtained comprises data selected from the group of roll angle data, pitch angle data, yaw angle data, and positional data, for use in connection with operation of a user-interaction interface for a computerized device.

9. The method of claim 2 wherein:

said each said modulated structured light pattern was obtained, prior to said step of projecting, by modulation using a distinct signal waveform;

said step of performing a demodulation comprises performing said demodulation along said orthogonal dimension; and

further comprising the step of displaying said reconstructed surface.

10. The method of claim 9 wherein:

said projected composite image comprises electromagnetic energy radiated within a wavelength range from 180 nm through 2000 nm;

said step of capturing a reflected image comprises receiving and recording pixel intensity and pixel wavelength of the energy reflected from the surface; and

said step of displaying further comprises displaying the object in three-dimensional space.

11. The method of claim 1 further comprising, after said step of capturing said first reflected image, the steps of:

projecting a second composite image comprising a second plurality of modulated structured light patterns, at the object;

capturing a second reflected image;

recovering second pattern information from said second reflected image, for each of said second plurality of modulated structured light patterns; and

producing a depth map of the surface using said recovered pattern information for said first and second plurality of modulated structured light patterns.

12. The method of claim 1 further comprising the steps of:

projecting a second composite image comprising a second plurality of modulated structured light patterns, at the object, said projection of said second composite image oriented such that orthogonal and phase dimensions of said second composite image are unaligned from orthogonal and phase dimensions of said first composite image;

capturing a second reflected image; and

recovering pattern information from said second reflected image, for each of said second plurality of modulated structured light patterns.

13. A method for retrieving depth information about at least one surface of an object, comprising the steps of:

projecting a composite image comprising a plurality of modulated structured light patterns, at the object;

capturing an image reflected from the surface;

recovering pattern information from said reflected image, for each of said plurality of modulated structured light patterns;

projecting a second composite image comprising a second plurality of modulated structured light patterns, at the object, said first and second composite images projected using, respectively, electromagnetic energy radiated within a first and a second wavelength range, said first and second wavelength ranges having no overlap;

capturing a second reflected image; and

recovering pattern information from said second reflected image, for each of said second plurality of modulated structured light patterns.

14. A system for retrieving depth information about at least one surface of an object under observation by at least one camera, the system comprising:

at least one projector adapted for projecting at least one composite image at the object, said composite image comprising a plurality of modulated structured light patterns, each having been modulated using a respective carrier frequency along the orthogonal dimension;

the camera adapted for capturing an image being reflected from the surface; and

a processor adapted for recovering pattern information from said reflected image, for each said plurality of modulated structured light patterns;

the depth information retrievable from said recovered pattern information.

15. The system of claim 14 wherein each said modulated structured light pattern was obtained by modulation with a distinct signal waveform associated with said respective carrier frequency;

said processor is further adapted to recover said pattern information by performing a demodulation along said orthogonal dimension of said reflected image; and

the depth information is used to produce a depth map of the surface.

16. The system of claim 15 wherein:

each said signal waveform used for said modulation of a respective structured light pattern, is distinct from each other of said signal waveforms used for said modulation of other structured light patterns; and

each said distinct signal waveform is selected from the group consisting of frequency shifted waveforms, phase shifted waveforms, random noise waveforms, and encoded waveforms.

17. The system of claim 15 wherein said processor is further adapted to compare said depth map with a plurality of stored depth map records of a feature for purposes of identification of the object.

18. The system of claim 15 wherein said processor is further adapted to obtain orientation data about the object from the depth map, said orientation data to comprise data selected from the group of roll angle data, pitch angle data, yaw angle data, and positional data, for use in connection with operation of a user-interaction interface for a device.

19. The system of claim 14 further comprising:

a second projector adapted for projecting, after said capturing an image, a second composite image comprising a second plurality of modulated structured light patterns, at the object;

a second camera adapted for capturing a second reflected image; and

said processor is further adapted to produce a depth map of the surface using (a) said recovered pattern information for said first plurality of modulated structured light patterns, and

(b) pattern information recovered for each of said second plurality of modulated structured light patterns.

20. The system of claim 14 further comprising:

a second projector adapted for projecting a second composite image comprising a second plurality of modulated structured light patterns, at the object, said projection of said second composite image oriented such that orthogonal and phase dimensions of said second composite image are unaligned from orthogonal and phase dimensions of said first composite image; and

a second camera adapted for capturing a second reflected image.

21. A system for retrieving depth information about at least one surface of an object under observation by at least one camera, the system comprising:

at least one projector adapted for projecting at least one composite image at the object, said composite image comprising a plurality of modulated structured light patterns;

the camera adapted for capturing an image being reflected from the surface;

a processor adapted for recovering pattern information from said reflected image, for each said plurality of modulated structured light patterns;

the depth information retrievable from said recovered pattern information;

a second projector adapted for projecting a second composite image comprising a second plurality of modulated structured light patterns, at the object, said first and second composite images being projected using, respectively, electromagnetic energy radiated within a first and a second wavelength range, said first and second wavelength ranges having no overlap; and

a second camera adapted for capturing a second reflected image.

22. A computer readable storage medium having stored thereon, program code for retrieving depth information about at least one surface of an object under observation by at least one camera, the program code comprising:

a first program sub-code for modulating each of a plurality of structured light patterns using a respective carrier frequency along the orthogonal dimension and projecting a composite image comprising said plurality of modulated structured light patterns, at the object;

a second program sub-code for recovering pattern information from an image reflected from the surface, for each said plurality of modulated structured light patterns; and

a third program sub-code for reconstructing the surface using said recovered pattern information.

23. The program code of claim 22 further comprising:

a fourth program sub-code for projecting, after said recovering pattern information, a second composite image comprising a second plurality of modulated structured light patterns, at the object; and

a fifth program sub-code for producing a depth map of the surface using (a) said recovered pattern information for said first plurality of modulated structured light patterns, and (b) pattern information recovered for each of said second plurality of modulated structured light patterns.

24. The program code of claim 22 further comprising:

a fourth program sub-code for projecting a second composite image comprising a second plurality of modulated structured light patterns, at the object;

said fourth program sub-code comprising instructions for projection of said second composite image oriented such that orthogonal and phase dimensions of said second composite image are unaligned from orthogonal and phase dimensions of said first composite image.

25. The storage medium of claim 22 wherein:

said first program sub-code comprises instructions for modulating each said plurality of structured light patterns with a distinct signal waveform associated with said respective carrier frequency;

said second program sub-code comprises instructions for performing a demodulation of said reflected image to produce a recovered pattern information for each of said plurality of modulated structured light patterns; and

said third program sub-code comprises instructions for producing a depth map of the surface using said recovered pattern information.

26. The storage medium of claim 25 wherein the program code further comprises a fourth program sub-code for comparing said depth map with a plurality of stored depth map records of a feature for purposes of identification of the object.

27. The storage medium of claim 25 wherein the program code further comprises a fourth program sub-code for obtaining orientation data about the object from the depth map, said orientation data to comprise data selected from the group of roll angle data, pitch angle data, yaw angle data, and positional data, for use in connection with operation of a user-interaction interface for a computerized device.

28. A computer readable storage medium having stored thereon, program code for retrieving depth information about at least one surface of an object under observation by at least one camera, the program code comprising:

a first program sub-code for projecting a composite image comprising a plurality of modulated structured light patterns, at the object;

a second program sub-code for recovering pattern information from an image reflected from the surface, for each said plurality of modulated structured light patterns;

a third program sub-code for reconstructing the surface using said recovered pattern information; and

a fourth program sub-code for projecting a second composite image comprising a second plurality of modulated structured light patterns, at the object;

said fourth program sub-code comprising instructions for projecting said first and second composite images using, respectively, electromagnetic energy radiated within a first and a second wavelength range, said first and second wavelength ranges having no overlap.