Dynamic generation of imperceptible structured light for tracking and acquisition of three dimensional scene geometry and surface characteristics in interactive three dimensional computer graphics applications

• US 5,870,136 A
• Filed: 12/05/1997
• Issued: 02/09/1999
• Est. Priority Date: 12/05/1997
• Status: Expired due to Fees

• Current Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Original Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Inventors: Livingston, Mark Alan, Welch, Gregory Francis, Bishop, Thomas Gary, Fuchs, Henry
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US08/986,309
• Time in Patent Office: 431 Days
• Field of Search: 348/36, 348/37, 348/38, 348/39, 348/40, 348/121, 348/135, 348/123, 348/42, 348/169, 382/106, 382/154, 356/376
• US Class Current: 348/42
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Methods, systems and computer products are provided for tracking objects within a scene using imperceptible structured light. The imperceptible structured light may be used in environments where humans work and therefore avoids disorienting observers. The structured light patterns are generated dynamically, allowing tracking without physical landmarks. Moreover, the occlusion of the generated landmarks is overcome by determining the position of objects using a determined camera position. The imperceptible structured light may be used in systems without tracking. The tracking may be used without imperceptible structured light.

Claims

1. A method for tracking the movement of a camera within the geometry of a scene using structured light comprising the steps of:

projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein a first structured light pattern is structured so as to detect geometric data of the scene and is projected for a first projection time onto the scene and a second structured light pattern is structured so as to reduce an observer'"'"'s perception of a visual artifact created by the projection of the first structured light pattern and is projected for a second projection time;

detecting a reflected light pattern from the scene in a two dimensional image plane of the camera, wherein the reflected light pattern is a reflection of the first structured light pattern;

sampling the scene coincident with the projection of the first structured light pattern; and

determining a camera location based on a correspondence between the position of elements of the first structured light pattern and the position of the reflection of the same elements in a camera image plane.

2. The method of claim 1, wherein said projecting step comprises the step of projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein the structure of the second pattern is selected so as to project an amount of light that complements the light projected onto the scene by the first pattern to maintain a substantially uniform density of light in the scene over the projection interval.

3. The method of claim 1, wherein said determining step comprises the steps of:

predicting a line equation representing a projection of a segment of the first structured light pattern within the two dimensional image plane of a camera;

determining an error associated with the predicted line equation and the position of the reflected light pattern in the two dimensional image plane; and

applying the error associated so as to update a previous estimated location of the camera so as to produce a current estimated camera position.

4. A method for determining the geometry of a scene comprising the steps of:

projecting a sequence of at least two structured patterns within a projection interval onto a scene, wherein a first structured light pattern is structured so as detect geometric data of the scene and is projected for a first projection time onto the scene and a second structured light pattern is structured so as to reduce an observer'"'"'s perception of a visual artifact created by the projection of the first structured light pattern and is projected for a second projection time; and

examining the scene coincident with the projection of the first structured light pattern so as to determine three dimensional information about the scene from a reflected pattern of the first structured light pattern.

5. The method of claim 4, wherein said projecting step comprises the step of:

projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein the structure of the second structured light pattern is selected so as to project an amount of light that complements the light projected onto the scene by the first structured light pattern so as to maintain a substantially uniform density of light in the scene over the projection.

6. The method of claim 5, wherein said projecting step comprises the steps of:

projecting a first structured light pattern onto a scene; and

projecting a second structured light pattern onto a scene, wherein the second structured light pattern represents a negative image of the first structured light pattern.

7. The method of claim 6, wherein the sequence of structured light patterns are generated by at least one pattern generator.

8. The method of claim 4, wherein said step of examining comprises the steps of:

generating a timing signal so as to be coincident with the projection of the first structured light pattern; and

applying the timing signal to a camera to make the camera exposure coincident with the projection of the first structured light pattern onto the scene.

9. The method of claim 4, further comprising the step of locating an object within the scene based on the determined three dimensional information about the scene and a camera location.

10. The method of claim 4, further comprising the step of locating the camera within the scene based on the determined three dimensional information about the scene and an object location.

11. The method claim 4, wherein said step of projecting comprises the step of projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein the first projection time is independent of the second projection time.

12. The method of claim 4, wherein said projecting step comprises the step of projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein the structure of the structured light pattern is defined using pixel intensity values.

13. The method of claim 4, wherein said projecting step comprises the step of projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein the structure of the first structured light pattern is different from the structure of the second structured light pattern.

14. A method for tracking the movement of a camera within the geometry of a scene comprising the steps of:

projecting a pattern onto a scene from a known position;

detecting a reflected pattern from the scene in the two dimensional image plane of at least one camera; and

determining a current estimated camera location based on a correspondence between the position of elements of the projected pattern and the position of the corresponding elements of the reflected pattern in the camera image plane.

15. The method of claim 14, wherein said determining step comprises the steps of:

predicting a line equation representing a projection of a segment of the projected pattern within the two dimensional image plane of a camera;

determining an error associated with the predicted line equation and the position of the reflected pattern in the two dimensional image plane; and

applying the error associated with the predicted line equation so as to update a previous estimated location of the camera to provide a current estimated camera position.

16. The method of claim 15, wherein said predicting step comprises the step of performing a transform of the projected pattern from projector space to camera space.

17. The method of claim 15, wherein said determining step comprises the step determining the displacement between the predicted line equation and the position of the reflected pattern in the camera'"'"'s two dimensional image plane.

18. The method of claim 15, wherein said applying step comprises the step of performing a transform from a previous estimated camera position to a current estimated camera position.

19. The method of claim 15, wherein said applying step comprises the step of applying the error associated with the predicted line equation so as to update a previous estimated location of the camera so as to provide a current estimated camera position, wherein the camera position is defined by three variables for a location within the scene and three variables for a camera pose.

20. The method of claim 14, further comprising the step of locating a position of an object within the scene based on a current estimated camera location and a known projector position and the projected pattern and the reflected pattern.

21. The method of claim 14, wherein said step of determining comprises the steps of

determining a correspondence between the elements within the projected pattern and elements within the detected pattern by analyzing the location of each element in the projected pattern and the detected pattern.

24. The system of claim 21, wherein said means for determining comprises the steps of:

means for predicting a line equation representing a projection of a segment of the first structured light pattern within the two dimensional image plane of a camera;

means for determining an error associated with the predicted line equation and the position of the reflected light pattern in the two dimensional image plane; and

means for applying the error associated so as to update a previous estimated location of the camera so as to produce a current estimated camera position.

22. A system for tracking the movement of a camera within the geometry of a scene using structured light comprising:

means for projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein a first structured light pattern is structured so as to detect geometric data of the scene and is projected for a first projection time onto the scene and a second structured light pattern is structured so as to reduce an observer'"'"'s perception of a visual artifact created by the projection of the first structured light pattern and is projected for a second projection time;

means for detecting a reflected light pattern from the scene in a two dimensional image plane of the camera, wherein the reflected light pattern is a reflection of the first structured light pattern;

means for sampling the scene coincident with the projection of the first structured light pattern; and

means for determining a camera location based on a correspondence between the position of elements of the first structured light pattern and the position of the reflection of the same elements in a camera image plane.

23. The system of claim 22, wherein said means for projecting comprises the step of projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein the structure of the second pattern is selected so as to project an amount of light that complements the light projected onto the scene by the first pattern to maintain a substantially uniform density of light in the scene over the projection interval.

25. A system for determining the geometry of a scene comprising:

means for projecting a sequence of at least two structured patterns within a projection interval onto a scene, wherein a first structured light pattern is structured so as detect geometric data of the scene and is projected for a first projection time onto the scene and a second structured light pattern is structured so as to reduce an observer'"'"'s perception of a visual artifact created by the projection of the first structured light pattern and is projected for a second projection time; and

means for examining the scene coincident with the projection of the first structured light pattern so as to determine three dimensional information about the scene from a reflected pattern of the first structured light pattern.

26. The system of claim 25, further comprising means for locating an object within the scene based on

the determined three dimensional information about the scene and a camera location.

27. The system of claim 25, wherein said means for projecting comprises:

means for projecting a sequence of at least two structured light patterns within a projection interval onto a scene, wherein the structure of the second structured light pattern is selected so as to project an amount of light that complements the light projected onto the scene by the first structured light pattern so as to maintain a substantially uniform density of light in the scene over the projection.

28. A system for tracking the movement of a camera within the geometry of a scene comprising:

means for projecting a pattern onto a scene from a known position;

means for detecting a reflected pattern from the scene in the two dimensional image plane of at least one camera; and

means for determining a current estimated camera location based on a correspondence between the position of elements of the projected pattern and the position of the corresponding elements of the reflected pattern in the camera image plane.

29. The system of claim 28, wherein said means for determining comprises:

means for predicting a line equation representing a projection of a segment of the projected pattern within the two dimensional image plane of a camera;

means for determining an error associated with the predicted line equation and the position of the reflected pattern in the two dimensional image plane; and

means for applying the error associated with the predicted line equation so as to update a previous estimated location of the camera to provide a current estimated camera position.