3D geometric modeling and 3D video content creation

• US 9,367,952 B2
• Filed: 05/20/2015
• Issued: 06/14/2016
• Est. Priority Date: 11/21/2006
• Status: Active Grant

• Current Assignee: Mantis Vision, Ltd.
• Original Assignee: Mantis Vision, Ltd.
• Inventors: Gordon, Eyal, Bittan, Gur Arie
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US14/717,487
• Publication Number:
  US 20150262411A1
• Time in Patent Office: 391 Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings

Abstract

A system, apparatus and method of obtaining data from a 2D image in order to determine the 3D shape of objects appearing in said 2D image, said 2D image having distinguishable epipolar lines, said method comprising: (a) providing a predefined set of types of features, giving rise to feature types, each feature type being distinguishable according to a unique bi-dimensional formation; (b) providing a coded light pattern comprising multiple appearances of said feature types; (c) projecting said coded light pattern on said objects such that the distance between epipolar lines associated with substantially identical features is less than the distance between corresponding locations of two neighboring features; (d) capturing a 2D image of said objects having said projected coded light pattern projected thereupon, said 2D image comprising reflected said feature types; and (e) extracting: (i) said reflected feature types according to the unique bi-dimensional formations; and (ii) locations of said reflected feature types on respective epipolar lines in said 2D image.

Claims

1. A device, comprising:

a memory configured to store a digital image of a reflected portion of a projected structured light pattern that is comprised of a repeating structure of a plurality of unique feature types;

a processor configured to identify and locate feature types appearing in the image and using known locations of each one of a plurality of distinguishable epipolar lines on a sensor'"'"'s plane and a constraint that any feature type appears at most once along any one of the plurality of distinguishable epipolar lines, wherein a distance between the distinguishable epipolar lines is less than YC, where C is a size of a feature type in the image in the shortest dimension of the repeating structure, and Y is a number of feature types in the shortest dimension of the repeating structure; and

the processor is configured to extract depth information based on locations of the feature types along respective distinguishable epipolar lines.

2. The device according to claim 1, wherein the repeating structure of the plurality of feature types embodies a coding matrix.

3. The device according to claim 1, wherein the repeating structure of the plurality of feature types is a tile.

4. The device according to claim 1 wherein the repeating structure of the plurality of feature types is a periodic repeating structure.

5. The device according to claim 1, wherein each one of the plurality of unique feature types is characterized by a unique combination of spatial formations.

6. The device according to claim 1, further comprising:

a projector configured to project a coded pattern, such that when the projector projects light, the structured light pattern is projected; and

a sensor configured to capture the digital image of the reflected portion of the projected structured light pattern, wherein the locations of the distinguishable epipolar lines on the sensor'"'"'s plane are associated with a spatial relation between the projector and the sensor,

and wherein said projector is orientated with respect to the sensor such that when the projector projects, the projected structured light pattern is at an angle relative to an epipolar field associated with the spatial relation between the projector and the sensor, and wherein the distance between the distinguishable epipolar lines is associated with an orientation of the projected structured light pattern relative to the epipolar field.

7. The device according to claim 6, wherein the projector is configured, such that when the projector projects the structured light pattern, an orientation of the structured light pattern with respect to the epipolar field is limited by a limiting epipolar separation factor.

8. The device according to claim 6, wherein the projector is configured such that when the projector projects, the projected structured light pattern is at an angle θ

relative to an epipolar field, where θ

=Sin<sup>−

1</sup>(P/YC), P denotes a distance by which distinguishable epipolar lines are separated, C is a size in the shortest dimension of the repeating structure of a feature type in the image, and Y is a number of feature types in the shortest dimension of the repeating structure.

9. The device according to claim 1, wherein the device has a predefined operating range, and in the image, two or more repetitions of a feature type appear on a distinguishable epipolar line, but within any given section of an epipolar line which corresponds to the operating range of the device, any feature type appears at most once.

10. The device according to claim 9, wherein the processor is configured to extract depth information based on a location in a given section of a distinguishable epipolar line of a given feature type on the distinguishable epipolar line.

11. The device according to claim 1, wherein Y equals 1.

12. The device according to claim 1, further comprising:

a sensor configured to capture the digital image of the reflected portion of the projected structured light pattern;

and a projector, the projector comprising;

a patterning element; and

a light source,

and wherein the patterning element is configured, such that when the light source illuminates the patterning element, the patterning element manipulates light from the light source to provide the projected structured light pattern, and wherein the patterning element is orientated with respect to the sensor such that when the projector projects the structured light pattern, the projected structured light pattern is at an angle relative to an epipolar field associated with the spatial relation between the projector and the sensor.

13. A method, comprising:

obtaining a digital image of a reflected portion of a projected structured light pattern that is comprised of a repeating structure of a plurality of unique feature types;

identifying and locating feature types appearing in the image using known locations of each one of a plurality of distinguishable epipolar lines on a sensor'"'"'s plane and a constraint that any feature type appears at most once along any one of the plurality of distinguishable epipolar lines, wherein a distance between the distinguishable epipolar lines is less than YC, where C is a size of a feature type in the image in the shortest dimension of the repeating structure, and Y is a number of feature types in the shortest dimension of the repeating structure; and

extracting depth information based on locations of the feature types along respective distinguishable epipolar lines.

14. The method according to claim 13, wherein the repeating structure of the plurality of feature types embodies a coding matrix.

15. The method according to claim 13, wherein the repeating structure of the plurality of feature types is a tile.

16. The method according to claim 13, wherein the repeating structure of the plurality of feature types is a periodic repeating structure.

17. The method according to claim 13, wherein each one of the plurality of unique feature types is characterized by a unique combination of spatial formations.

18. The method according to claim 13, further comprising projecting the structured light pattern such that the projected structured light pattern is orientated, at an angle relative to an epipolar field associated with a spatial relation between a projector that is used to project the structured light pattern and a sensor that is used to capture the image, and wherein the distance between the distinguishable epipolar lines is associated with the orientation of the projected structured light pattern relative to the epipolar field.

19. The method according to claim 18, wherein the orientation of the projected structured light pattern with respect to the epipolar field is limited by a limiting epipolar separation factor.

20. The method according to claim 13, wherein in said image two or more repetitions of a feature type appear on a distinguishable epipolar line, but within any given section of an epipolar line which corresponds to an operating range of a depth sensing device that is used to provide the projected structured light pattern and to capture the digital image, any feature type appears at most once.

21. The method according to claim 20, wherein said extracting depth information comprises, extracting depth information based on a location in a given section of a distinguishable epipolar line of a given feature type on the distinguishable epipolar line.

22. The method according to claim 13, wherein Y equals 1.

23. The method according to claim 13, where said projecting comprises, projecting the structured light pattern at an angle θ

relative to an epipolar field, where θ

=Sin<sup>−

1</sup>(P/YC), P denotes a distance by which distinguishable epipolar lines are separated, C is a size in the shortest dimension of the repeating structure of a feature type in the image, and Y is a number of feature types in the shortest dimension of the repeating structure.

24. A non-transitory machine-readable storage medium including instructions which, when performed by one or more processors, causes the one or more processors to perform a method of extracting depth information from a scene, the method comprising:

obtaining a digital image of a reflected portion of a projected structured light pattern that is comprised of a repeating structure of a plurality of unique feature types;

identifying and locating feature types appearing in the image using known locations of each one of a plurality of distinguishable epipolar lines on a sensor'"'"'s plane and a constraint that any feature type appears at most once along any one of the plurality of distinguishable epipolar lines, wherein a distance between the distinguishable epipolar lines is less than YC, where C is a size of a feature type in the image in the shortest dimension of the repeating structure, and Y is a number of feature types in the shortest dimension of the repeating structure; and

extracting depth information based on locations of the feature types along respective distinguishable epipolar lines.

25. The method according to claim 24, wherein the repeating structure of the plurality of feature types embodies a coding matrix.

26. The method according to claim 24, wherein the repeating structure of the plurality of feature types is a tile.

27. The method according to claim 24, wherein each one of the plurality of unique feature types is characterized by a unique combination of spatial formations.

28. The method according to claim 24, wherein in said image, two or more repetitions of a feature type appear on a distinguishable epipolar line, but within any given section of an epipolar line which corresponds to an operating range of a depth sensing device that is used to provide the projected structured light pattern and to capture the digital image, any feature type appears at most once.