SYSTEM AND METHOD FOR AUTOMATIC ALIGNMENT AND PROJECTION MAPPING

• US 20170039756A1
• Filed: 08/07/2015
• Published: 02/09/2017
• Est. Priority Date: 08/07/2015
• Status: Active Grant

• Current Assignee: Christie Digital Systems USA Incorporated (Ushio Incorporated)
• Original Assignee: Christie Digital Systems USA Incorporated (Ushio Incorporated)
• Inventors: MOULE, Kevin, SCOTT, Derek, WASILKA, Nick, FRASER, Ian James, ROBINSON, James
• Granted Patent: US 10,089,778 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2504 Calibration devices
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 5/74 Projection arrangements for image reproduction, e.g. using eidophor

Abstract

A system and method for automatic alignment and projection mapping are provided. A projector and at least two cameras are mounted with fields of view that overlap a projection area on a three-dimensional environment. A computing device: controls the projector to project structured light patterns that uniquely illuminate portions of the environment; acquires images of the patterns from the cameras; generates a two-dimensional mapping of the portions between projector and camera space and by processing the images and correlated patterns; generates a cloud of points representing the environment using the mapping and camera positions; determines a projector location, orientation and lens characteristics from the cloud; positions a virtual camera relative to a virtual three-dimensional environment, corresponding to the environment, parameters of the virtual camera respectively matching parameters of the projector; and, controls the projector to project based on a virtual location, orientation and characteristics of the virtual camera.

Claims

1. A system comprising:

a computing device;

a projector; and

at least two cameras, each of the projector and the at least two cameras mounted relative to a three-dimensional environment with respective fields of view at least partially overlapping a projection area of the projector on the three-dimensional environment;

the computing device configured to;

control the projector to sequentially project one or more structured light patterns configured to uniquely illuminate different portions of the three-dimensional environment;

acquire one or more respective images from each of the at least two cameras while the projector is projecting the one or more structured light patterns, each of the one or more respective images correlated with a given respective structured light pattern;

generate a two-dimensional mapping of the different portions of the three-dimensional environment between a projector space and a camera space by processing the respective images and correlated given respective structured light patterns;

generate a cloud of points representing the three-dimensional environment using the two-dimensional mapping and given positions of the at least two cameras relative to the three-dimensional environment;

determine a location, an orientation and lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points;

position a virtual camera relative to a virtual three-dimensional environment, corresponding to the three-dimensional environment, a virtual location, a virtual orientation and virtual lens characteristics of the virtual camera respectively matching the location, the orientation and the lens characteristics of the projector; and

,

control the projector to project based on the virtual location, the virtual orientation and the virtual lens characteristics of the virtual camera.

2. The system of claim 1, wherein the three-dimensional environment comprises a physical object, and the system further comprises a memory storing a virtual model of the physical object, the computing device further configured to:

determine a respective location and respective orientation of the physical object relative to a physical origin of the three-dimensional environment by comparing the virtual model to the cloud of points;

set a virtual location and virtual orientation of the virtual model in the virtual three-dimensional environment with respect to a virtual origin, related to the physical origin, to match the respective location and the respective orientation of the physical object; and

,

control the projector to illuminate the physical object with images adjusted for the virtual location and the virtual orientation of the virtual model.

3. The system of claim 2, wherein the comparing the virtual model to the cloud of points comprises an iterative closest point scheme.

4. The system of claim 1, wherein the three-dimensional environment comprises one or more of:

a physical object on a surface;

a curved surface; and

an irregular surface.

5. The system of claim 1, wherein the one or more structured light patterns comprises a sequence of structured light patterns configured to uniquely sequentially illuminate different portions of the three-dimensional environment.

6. The system of claim 1, wherein the one or more structured light patterns comprise a sequence of images of horizontal bars and vertical bars, the bars either halving or doubling in width within the sequence and in each direction.

7. The system of claim 1, wherein the one or more structured light patterns comprises one or more of:

a sequence of images of bars;

black and white images;

colored images; and

images of sinusoids.

8. The system of claim 1, wherein the computing device is further configured to determine the given positions of the at least two cameras relative to the three-dimensional environment, and set a physical origin of the three-dimensional environment by:

acquiring at least one image of a known target pattern positioned in the three-dimensional environment from each of the at least two cameras; and

,

comparing the at least one image of the known target pattern with a representation of the known target pattern.

9. The system of claim 1, wherein the computing device is further configured to generate the cloud of points representing the three-dimensional environment using the two-dimensional mapping and the given positions of the at least two cameras relative to the three-dimensional environment using triangulation.

10. The system of claim 1, wherein the computing device is further configured to determine the location, the orientation and the lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points by matching a plurality of points from the cloud of points to respective areas of one or more of the structured light patterns.

11. The system of claim 1, wherein the virtual location, the virtual orientation and the virtual lens characteristics comprises pose data comprising:

a three-dimensional position of the virtual camera in the virtual three-dimensional environment with respect to a virtual origin, related to the physical origin;

a pitch value;

a yaw value;

a roll value;

an offset value; and

a throw value.

12. A method comprising:

in a system comprising;

a computing device;

a projector; and

at least two cameras, each of the projector and the at least two cameras mounted relative to a three-dimensional environment with respective fields of view at least partially overlapping a projection area of the projector on the three-dimensional environment, controlling the projector, using the computing device, to sequentially project one or more structured light patterns configured to uniquely illuminate different portions of the three-dimensional environment;

acquiring one or more respective images from each of the at least two cameras, using the computing device, while the projector is projecting the one or more structured light patterns, each of the one or more respective images correlated with a given respective structured light pattern;

generating, using the computing device, a two-dimensional mapping of the different portions of the three-dimensional environment between a projector space and a camera space by processing the respective images and correlated given respective structured light patterns;

generating, using the computing device, a cloud of points representing the three-dimensional environment using the two-dimensional mapping and given positions of the at least two cameras relative to the three-dimensional environment;

determining, using the computing device, a location, an orientation and lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points;

positioning, using the computing device, a virtual camera relative to a virtual three-dimensional environment, corresponding to the three-dimensional environment, a virtual location, a virtual orientation and virtual lens characteristics of the virtual camera respectively matching the location, the orientation and the lens characteristics of the projector; and

,

controlling the projector, using the computing device, to project based on the virtual location, the virtual orientation and the virtual lens characteristics of the virtual camera.

13. The method of claim 12, wherein the three-dimensional environment comprises a physical object, and the system further comprises a memory storing a virtual model of the physical object, the method further comprising:

determining a respective location and respective orientation of the physical object relative to a physical origin of the three-dimensional environment by comparing the virtual model to the cloud of points;

setting a virtual location and virtual orientation of the virtual model in the virtual three-dimensional environment with respect to a virtual origin, related to the physical origin, to match the respective location and the respective orientation of the physical object; and

,

controlling the projector to illuminate the physical object with images adjusted for the virtual location and the virtual orientation of the virtual model.

14. The method of claim 13, wherein the comparing the virtual model to the cloud of points comprises an iterative closest point scheme.

15. The method of claim 12, wherein the one or more structured light patterns comprises one or more of:

a sequence of structured light patterns configured to uniquely sequentially illuminate different portions of the three-dimensional environment;

a sequence of images of horizontal bars and vertical bars, the bars either halving or doubling in width within the sequence and in each direction;

a sequence of images of bars;

black and white images;

colored images; and

images of sinusoids.

16. The method of claim 12, further comprising determining the given positions of the at least two cameras relative to the three-dimensional environment, and set a physical origin of the three-dimensional environment by:

acquiring at least one image of a known target pattern positioned in the three-dimensional environment from each of the at least two cameras; and

,

comparing the at least one image of the known target pattern with a representation of the known target pattern.

17. The method of claim 12, further comprising generating the cloud of points representing the three-dimensional environment using the two-dimensional mapping and the given positions of the at least two cameras relative to the three-dimensional environment using triangulation.

18. The method of claim 12, further comprising determining the location, the orientation and the lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points by matching a plurality of points from the cloud of points to respective areas of one or more of the structured light patterns.

19. The method of claim 12, wherein the virtual location, the virtual orientation and the virtual lens characteristics comprises pose data comprising:

a three-dimensional position of the virtual camera in the virtual three-dimensional environment with respect to a virtual origin, related to the physical origin;

a pitch value;

a yaw value;

a roll value;

an offset value; and

a throw value.

20. A non-transitory computer-readable medium storing a computer program, wherein execution of the computer program is for:

in system comprising;

a computing device;

a projector; and

at least two cameras, each of the projector and the at least two cameras mounted relative to a three-dimensional environment with respective fields of view at least partially overlapping a projection area of the projector on the three-dimensional environment, controlling the projector, using the computing device, to sequentially project one or more structured light patterns configured to uniquely illuminate different portions of the three-dimensional environment;

acquiring one or more respective images from each of the at least two cameras, using the computing device, while the projector is projecting the one or more structured light patterns, each of the one or more respective images correlated with a given respective structured light pattern;

generating, using the computing device, a two-dimensional mapping of the different portions of the three-dimensional environment between a projector space and a camera space by processing the respective images and correlated given respective structured light patterns;

generating, using the computing device, a cloud of points representing the three-dimensional environment using the two-dimensional mapping and given positions of the at least two cameras relative to the three-dimensional environment;

determining, using the computing device, a location, an orientation and lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points;

positioning, using the computing device, a virtual camera relative to a virtual three-dimensional environment, corresponding to the three-dimensional environment, a virtual location, a virtual orientation and virtual lens characteristics of the virtual camera respectively matching the location, the orientation and the lens characteristics of the projector; and

,

controlling the projector, using the computing device, to project based on the virtual location, the virtual orientation and the virtual lens characteristics of the virtual camera.