Coded-array technique for obtaining depth and other position information of an observed object

• US 7,212,663 B2
• Filed: 06/19/2003
• Issued: 05/01/2007
• Est. Priority Date: 06/19/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Tomasi, Carlo
• Primary Examiner(s): Mancuso; Joseph
• Assistant Examiner(s): LIEW, ALEX KOK SOON
• Application Number:
  US10/600,744
• Publication Number:
  US 20040005092A1
• Time in Patent Office: 1,412 Days
• Field of Search: None
• US Class Current: 382/154
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50

Abstract

A projection array is provided that comprises a plurality of discrete projection elements. An image array is obtained of a scene with the light projected onto it that is coded using the projection array. Correspondence information is determined for each element in the image array, where the correspondence information can be used to determine which of the plurality of elements in the projection array corresponds to a particular image element. The determination of correspondence information for each element in the image array can be made independently of correspondence information for other elements in the image array.

Claims

1. A method for detecting characteristics of an object in a scene, the method comprising:

projecting a projection array, comprising a plurality of projection elements arranged in rows and columns, to project onto said scene and object coded light creating a pattern of said projection elements having a detectable characteristic that identifies a column in said projection array and belongs to a characteristic set of at least two characteristics, said pattern of projection elements being created from said projection array and defining a repeated pattern of staggered said columns;

capturing an image of said scene with said coded light projected onto said scene;

identifying a plurality of image elements forming an image array from a captured said image; and

determining for each image element in a captured said image correspondence information useable to determine which projection element in said plurality of projection elements corresponds to that image element, wherein determining correspondence information for each element is performed independently of knowing correspondence information for any other image element.

2. The method of claim 1, further comprising determining position information for each identified image element based at least in part on correspondence information for said identified image element.

3. The method of claim 1, further comprising:

providing a protector to protect said projection array;

providing a camera to capture said image array; and

determining position information for each identified image element based at least in part on correspondence information for said identified image element and on a position of said projector relative to said camera.

4. The method of claim 1, wherein determining correspondence information includes determining a row coordinate and a column coordinate in said image array for at least one said image element based at least in part on a row coordinate and a column coordinate in said projection array.

5. The method of claim 1, wherein determining correspondence information includes:

determining a row coordinate and a column coordinate in said image array for at least one said image element; and

for each at least one said image element, determining a row coordinate and a column coordinate for an element in said projection array corresponding to said image element.

6. The method of claim 1, wherein determining correspondence information is performed absent prior determination of correspondence information for a calibration element in said plurality of image elements designated as calibrating correspondence information for all other image elements in said plurality of image elements.

7. The method of claim 1, wherein determining correspondence information is performed absent prior determination of correspondence information for an element in said plurality of image elements designated as being a first element in a sequence of elements that correspond substantially to an entire row of elements in said projection array.

8. The method of claim 1, wherein determining correspondence information includes identifying missing image elements by identifying at least one projection element lacking a corresponding image element in said image array.

9. The method of claim 8, further comprising compensating for missing image elements by using image elements congruent to said missing image elements.

10. The method of claim 1, wherein projecting a projection array is carried out with a projector and includes identifying data representing a tangible medium used to pattern light emitted from said projector.

11. The method of claim 1, further including:

providing a projector to protect said projection array; and

identifying a projection array includes identifying data representing a diffraction pattern for structuring light emitted from said projector.

12. The method of claim 1, wherein each characteristic in said characteristic set has a characteristic selected from a group consisting of (a) a geometric characteristic distinguishable from an other geometric characteristic in said set, (b) color from another color in said set, and (c) a shape distinguishable from another shape in said set.

13. The method of claim 1, further comprising:

determining a sequence of values that can identify each projection element in said projection array;

assigning a sequence value to each characteristic in said characteristic set; and

wherein projecting coded light includes projecting each projection element in projection array as a characteristic in said characteristic set so that said sequence can be at least partially reflected in said pattern based on sequence value assigned to each characteristic.

14. The method of claim 13, wherein assigning a sequence value includes assigning in a manner selected from a group consisting of (a) assigning a binary value, (b) assigning non-repeating binary values, (c) assigning using a non-N-repeating bit assignment formula, and (d) assigning a sequence subset to individual column coordinates so as to correspond elements in said projection array with elements in said image array.

15. The method of claim 1, wherein projecting coded light includes creating said pattern so that detectable characteristics identify columns in said projection array and so that each row in said projection array has a same set of columns.

16. The method of claim 1, further comprising using said staggered arrangement to determine which projection elements in said projection array lack a corresponding image element.

17. The method of claim 1, further comprising identifying rows in said image array that lack a same column to determine which projection elements in said projection array lack a corresponding image element.

18. The method of claim 1, wherein at least one method step is performed on at least one processor executing instructions stored on a computer-readable medium.

19. A system to detect characteristics of an object in a scene, the system comprising:

means for projecting a projection array, comprising a plurality of projection elements arranged in rows and columns, to project onto said scene and object coded light creating a pattern of said projecting elements having a detectable characteristic that identifies a column in said position array and belongs to a characteristic set of at least two characteristics, said pattern of projection elements being created from said projection array and defining a repeated pattern of staggered said columns;

means for capturing an image of said scene with said coded light projected onto said scene;

means for identifying a plurality of image elements forming an image array from a captured said image; and

means for determining for each image element in a captured said image information useable to determine which projection element in said plurality of protect elements corresponds to an image element, wherein determining correspondence information for each element is performed independently of knowing correspondence information for any other image element.

20. The system of claim 19, wherein said means for projecting includes a projector, and identifies data representing a tangible medium used to pattern light emitted from said projector.

21. The system of claim 19, wherein said means for projecting includes a projector and includes data representing a diffraction pattern for structuring light emitted by said projector.

22. The system of claim 19, a characteristic in said characteristic set has a characteristic selected from a group consisting of (a) a geometric characteristic distinguishable from an other geometric characteristic in said set, (b) a color from another color in said set, and (c) a shape distinguishable from an other shape in said set.

23. The system of claim 19, further including:

means for determining a sequence of values that can identify each projection element in said projection array;

means for assigning a sequence value to each characteristic in said characteristic set;

wherein said means for projecting projects coded light and projects each projection element in said projection array as a characteristic in said characteristic set so that said sequence can be at least partially reflected in said pattern based on sequence value assigned to each characteristic.

24. The system of claim 19, wherein a sequence value is assigned in a manner selected from a group consisting of (a) assigning a binary value, (b) assigning non-repeating binary values, (c) assigning using a non-N-repeating bit assignment formula, and (d) assigning a sequence subset to individual column coordinates so as to correspond elements in said projection array with elements in said image array.

25. The system of claim 19, wherein said staggered arrangement is used in determining which projection elements in said projection array lack a corresponding image element.

26. The system of claim 19, wherein said means for projecting projects coded light and creates said pattern so that detectable characteristics identify columns in said projection array and so that each row in said projection array has a same set of columns.

27. The system of claim 19, wherein said means for determining determines position information for each identified image element based at least in part on correspondence information for said identified image element.

28. The system of claim 19, wherein:

said means for identifying a projection array includes a projector;

said means for obtaining an image includes a camera; and

said means for determining determines position information for each identified image element based at least in part on correspondence information for said identified image element and on a position of a said protector relative to said camera.

29. The system of claim 19, wherein said means for determining determines a row coordinate and a column coordinate in said image array for at least one said image element based at least in part on a row coordinate and a column coordinate in said projection array.

30. The system of claim 19, wherein said means for determining determines a row coordinate and a column coordinate in said image array for at least one said image element;

and

for each at least one said image element, determines a row coordinate and a column coordinate for an element in said projection array corresponding to said image element.

31. The system of claim 19, wherein said means for determining functions absent prior determination of correspondence information for a calibration element in said plurality of image elements designated as calibrating correspondence information for all other image elements in said plurality of image elements.

32. The system of claim 19, wherein said means for determining functions without prior determination of correspondence information for an element in said plurality of image elements designated as being a first element in a sequence of elements that correspond substantially to an entire row of elements in said projection array.

33. The system of claim 19, wherein said means for determining identifies missing image elements by identifying at least one projection element lacking a corresponding image element in said image array.

34. The system of claim 33, wherein said means for determining compensates for missing image elements by using image elements congruent to said missing image elements.

35. The system of claim 19, further including identifying rows in said image array that lack a same column to determine which projection elements in said projection array lack a corresponding image element.

36. The system of claim 19, further including identifying rows in said image array that lack a same column to determine which projection elements in said projection array lack a corresponding image element.

37. A system to detect characteristics of an object in a scene, the system comprising:

a projector to project a projection array, comprising a plurality of projection elements arranged in rows and columns, and to project onto said scene and object coded light creating a pattern of said projecting elements having a detectable characteristic that identifies a column in said position array and belongs to a characteristic set of at least two characteristics, said pattern of projection elements being created from said projection array and defining a repeated pattern of staggered said columns;

a camera to capture an image of said scene with said coded light projected onto said scene;

means for identifying a plurality of image elements forming an image array from a captured said image; and

means for determining for each image element in a captured said image information useable to determine which projection element in said plurality of project elements corresponds to an image element, wherein determining correspondence information for each element is performed independently of knowing correspondence information for any other image element.