3D geometric modeling and 3D video content creation

• US 10,140,753 B2
• Filed: 06/13/2016
• Issued: 11/27/2018
• Est. Priority Date: 11/21/2006
• Status: Active Grant

• Current Assignee: Mantis Vision, Ltd.
• Original Assignee: Mantis Vision, Ltd.
• Inventors: Gordon, Eyal, Bittan, Gur Arie
• Primary Examiner(s): Rosario, Dennis
• Application Number:
  US15/180,116
• Publication Number:
  US 20160364903A1
• Time in Patent Office: 897 Days
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings

Abstract

A system, apparatus and method of obtaining data from a 2D image in order to determine the 3D shape of objects appearing in said 2D image, said 2D image having distinguishable epipolar lines, said method comprising: (a) providing a predefined set of types of features, giving rise to feature types, each feature type being distinguishable according to a unique bi-dimensional formation; (b) providing a coded light pattern comprising multiple appearances of said feature types; (c) projecting said coded light pattern on said objects such that the distance between epipolar lines associated with substantially identical features is less than the distance between corresponding locations of two neighboring features; (d) capturing a 2D image of said objects having said projected coded light pattern projected thereupon, said 2D image comprising reflected said feature types; and (e) extracting: (i) said reflected feature types according to the unique bi-dimensional formations; and (ii) locations of said reflected feature types on respective epipolar lines in said 2D image.

Claims

1. A device for creating a 3D video stream providing 3D motion capture of an object, said device comprising:

a projector and a sensor having a fixed position in relation to each other;

said projector being configured to project a coded light pattern comprising a repeating structure of a plurality of unique feature types arranged such that each feature type appears at most once along any one of a plurality of distinguishable first epipolar lines; and

said sensor being configured to capture a series of images of reflections, upon the object, of a portion of the coded light pattern, the reflections being oriented at a non-zero angle in relation to second epipolar lines of an epipolar field in each captured image associated with the fixed relative position between said projector and said sensor, and

a processor configured to;

i. identify, within the reflections captured in the images, reflections of the feature types projected by said projector;

ii. calculate distances to points on the object based on the identified reflections;

iii. generate points of a series of point clouds using the calculated distances; and

iv. create a video stream providing 3D motion capture of the object based on the generated point clouds

wherein said projector is configured to;

i. project the coded light pattern at an angle to the fixed relative position between said projector and said sensor, and

ii. project the coded light pattern such that;

a. each one of the first epipolar lines has a known location in each captured image and is associated with the fixed relative position between the projector and the sensor, and

b. a distance between the second epipolar lines is less than YC, where C is a size of a feature type in each captured image in the shortest dimension of the repeating structure, and Y is a number of feature types in the shortest dimension of the repeating structure.

2. The device according to claim 1, wherein the repeating structure of the plurality of feature types embodies a coding matrix.

3. The device according to claim 1, wherein the repeating structure of the plurality of feature types is a tile.

4. The device according to claim 1 wherein the repeating structure of the plurality of feature types is a periodic repeating structure.

5. The device according to claim 1, wherein each one of the plurality of unique feature types is characterized by a unique combination of spatial formations.

6. The device according to claim 1, wherein the projector is configured, such that when the projector projects the coded light pattern, an orientation of the coded light pattern with respect to the epipolar field is limited by a limiting epipolar separation factor.

7. The device according to claim 1, wherein the device has a predefined operating range, and in each captured image, two or more repetitions of a feature type appear on a distinguishable epipolar line, but within any given section of an epipolar line which corresponds to an operating range of the device, any feature type appears at most once.

8. The device according to claim 7, further comprising a processor that is configured to extract depth information based on a location in a given section of the second epipolar lines of a given feature type on the second epipolar lines.

9. The device according to claim 1, wherein Y equals 1.

10. The device according to claim 1, wherein the projector is configured such that when the projector projects, a projected structured light pattern is at an angle θ

relative to an epipolar field, where θ

=Sin<sup>−

1</sup>(P/YC), P denotes a distance by which distinguishable epipolar lines are separated, C is the size in the shortest dimension of the repeating structure of a feature type in each captured image, and Y is the number of feature types in the shortest dimension of the repeating structure.

11. The device according to claim 1, wherein the coded light pattern is embodied in a patterning element.

12. The device according to claim 11, wherein the patterning element is a transparency mask, a diffractive optical element or an array of light emitting elements arranged to provide the plurality of unique feature types.

13. The device according to claim 11, wherein the projector and the patterning element are configured such that a light source is configured to illuminate the patterning element, and when the light source illuminates the patterning element, the patterning element manipulates light from the light source to provide the projected coded light pattern, and wherein the patterning element is orientated with respect to the sensor such that when the projector projects the coded light pattern, the projected coded light pattern is at an angle relative to an epipolar field associated with a spatial relation between the projector and the sensor that is used to capture each image.

14. A method for creating a 3D video stream providing 3D motion capture of an object, said method comprising:

projecting, using a projector, a coded light pattern comprising a repeating structure of a plurality of unique feature types arranged such that each feature type appears at most once along any one of a plurality of distinguishable first epipolar lines;

capturing, using a sensor having a fixed position in relation to the projector, a series of images of reflected reflections, upon the object, of a portion of the coded light pattern, the reflections being oriented at a non-zero angle in relation to second epipolar lines of an epipolar field in each captured image associated with the fixed relative position between said projector and said sensor,

using a processor functionally associated with the sensor to;

i. identify, within the reflections captured in the images, reflections of the feature types projected by said projector;

ii. calculate distances to points on the object based on the identified reflections;

iii. generate points of a series of point clouds using the calculated distances; and

iv. create a video stream providing 3D motion capture of the object based on the generated point clouds

wherein

the coded light pattern is projected at an angle to the fixed relative position between said projector and said sensor, and

each one of the first epipolar lines has a known location in each captured image and is associated with the fixed relative position between the projector and the sensor, and

a distance between the second epipolar lines is less than YC, where C is a size of a feature type in each captured image in the shortest dimension of the repeating structure, and Y is a number of feature types in the shortest dimension of the repeating structure.

15. The method according to claim 14, wherein the repeating structure of the plurality of feature types embodies a coding matrix.

16. The method according to claim 14, wherein the repeating structure of the plurality of feature types is a tile.

17. The method according to claim 14, wherein the repeating structure of the plurality of feature types is a periodic repeating structure.

18. The method according to claim 14, wherein each one of the plurality of unique feature types is characterized by a unique combination of spatial formations.

19. The method according to claim 14, wherein a patterning element used in the projection is orientated with respect to the sensor at an angle, and wherein a distance between the distinguishable first epipolar lines is associated with an orientation of the patterning element relative to the sensor.

20. The method according to claim 19, wherein the orientation of the patterning element relative to the sensor is limited by a limiting epipolar separation factor.

21. The method according to claim 14, further comprising obtaining an operating range of the sensor, and along any given section of an epipolar line which corresponds to a range that is less than or equal to the operating range, any feature type appears at most once, and wherein two or more repetitions of a feature type appear along an epipolar line section that corresponds to a range that is greater than the operating range.

22. The method according to claim 14, wherein Y equals 1.

23. The method according to claim 14, wherein when the projector projects the coded light pattern, the projected coded light pattern is at an angle θ

relative to the epipolar field, where θ

=Sin<sup>−

1</sup>(P/YC), P denotes the distance by which the distinguishable second epipolar lines are separated, C is the size in the shortest dimension of the repeating structure of a feature type in each captured image, and Y is the number of feature types in the shortest dimension of the repeating structure.