Super-resolution overlay in multi-projector displays

• US 20040239885A1
• Filed: 04/19/2004
• Published: 12/02/2004
• Est. Priority Date: 04/19/2003
• Status: Active Grant

• Current Assignee: University Of Kentucky Research Foundation (University of Kentucky)
• Original Assignee: University Of Kentucky Research Foundation (University of Kentucky)
• Inventors: Ramakrishnan, Divya T., Jaynes, Christopher O.
• Granted Patent: US 7,097,311 B2
• Time in Patent Office: Days
• US Class Current: 353/30
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026

Abstract

A technique, associated system and computer executable program code, for projecting a superimposed image onto a target display surface under observation of one or more cameras. A projective relationship between each projector being used and the target display surface is determined using a suitable calibration technique. A component image for each projector is then estimated using the information from the calibration, and represented in the frequency domain. Each component image is estimated by: Using the projective relationship, determine a set of sub-sampled, regionally shifted images, represented in the frequency domain; each component image is then composed of a respective set of the sub-sampled, regionally shifted images. In an optimization step, the difference between a sum of the component images and a frequency domain representation of a target image is minimized to produce a second, or subsequent, component image for each projector. Here, a second set of frequency domain coefficients for use in producing a frequency domain representation of the second component image for each projector is identified. Taking the inverse Fourier transform of the frequency domain representation of the second component image, converts the information into a spatial signal that is placed into the framebuffer of each component projector and projected therefrom to produce the superimposed image.

Claims

1. A method for projecting a superimposed image onto a target display surface under observation of at least one camera, the method comprising the steps of:

(a) determining a projective relationship between each of a plurality of projectors and the target display surface;

(b) estimating a component image for each said projector, comprising;

(1) using said projective relationship, for each of said component images determine a plurality of sub-sampled, regionally shifted images represented in the frequency domain; and

(2) composing each said component image using a respective plurality of said sub-sampled, regionally shifted images; and

(c) minimizing the difference between a sum of said component images and a frequency domain representation of a target image to produce a second component image for each said projector.

2. The method of claim 1 wherein said step of minimizing the difference between said sum and said frequency domain representation of said target image comprises:

(a) identifying a second set of frequency domain coefficients for use in producing a frequency domain representation of said second component image for a respective one of said projectors, and (b) taking the inverse Fourier transform of said frequency domain representation of said second component image for said respective one of said projectors; and

further comprising the step of projecting from each said projector, a respective one of said second component images to produce the superimposed image.

3. The method of claim 1 wherein said step of determining a projective relationship further comprises:

(a) observing the target display surface with the camera, c;

(b) recovering a homography between each projector, pi, and the camera, _Hpi^c, and

(c) using said homography, _Hpi^c, determine said projective relationship, _Hpi^b, wherein b represents a reference frame for the target display surface.

4. The method of claim 2 wherein said step of using said projective relationship to determine said plurality of sub-sampled, regionally shifted images, comprises:

(a) decomposing each said projective relationship, _Hpi^b, into a linear shift matrix representing offsets between a respective of said projectors pi, and said reference frame b; and

(b) using said linear shift matrix for said determining said plurality of sub-sampled, regionally shifted images associated with said respective of said projectors pi.

5. The method of claim 1 wherein said step of minimizing the difference between said sum and said frequency domain representation of said target image, _{FT[u, v]}, comprises:

(a) minimizing the expression;

${<br><br>F_T<br><br>\left[u,v\right]-\left(\underset{i=0}{\overset{k}{\sum <br><br>}}<br><br>F_{\mathrm{\Delta <br><br>}}^i<br><br>\left[u,v\right]\right)<br><br>}^2$

wherein k represents the total number of said plurality of projectors, said component image for each said projector is represented by

$F_{\mathrm{\Delta <br><br>}}^p<br><br>\left[u,v\right]=\underset{r=0}{\overset{N}{\sum <br><br>}}<br><br>F_{\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>r}<br><br>\left[u,v\right],$

and said sub-sampled, regionally shifted images represented in the frequency domain are represented by <sub>F Δ

r[u, v]</sub>, to identify a second set of frequency domain coefficients for use in producing a frequency domain representation of said second component image for a respective one of said projectors; and

(b) taking the inverse Fourier transform of said frequency domain representation of said second component image for said respective one of said projectors.

6. A system for projecting the superimposed image onto the target display surface, of claim 1, wherein:

being projected from each said projector, is a respective one of said second component images to produce the superimposed image.

7. A method for projecting a superimposed image onto a target display surface under observation of at least one camera, the method comprising the steps of:

(a) determining a projective relationship between each of a plurality of projectors and the target display surface;

(b) estimating a component image for each said projector, comprising;

(1) using said projective relationship, for each of said component images determine a plurality of sub-sampled, regionally shifted images represented in the frequency domain, and

(2) composing each said component image using a respective plurality of said sub-sampled, regionally shifted images; and

(c) performing an optimization using said component images and a frequency domain representation of a target image to produce a second component image for each said projector.

8. The method of claim 7 wherein:

(a) said step of performing an optimization comprises minimizing the difference between a sum of said component images and said frequency domain representation of said target image, _{FT[u, v]}, to produce said second component image for each said projector; and

(b) said step of determining a projective relationship further comprises;

(1) observing the target display surface with the camera, c;

(2) recovering a homography between each projector, pi, and the camera, _Hpi^c; and

(3) using said homography, _Hpi^b, determine said projective relationship, _Hpi^b, wherein b represents a reference frame for the target display surface.

9. The method of claim 8 wherein said step of performing an optimization comprises minimizing the difference between a sum of said component images and said frequency domain representation of said target image, _{FT[u, v]}, by minimizing the expression:

10. A system for projecting the superimposed image onto the target display surface, of claim 7, wherein:

being projected from each said projector, is a respective one of said second component images to produce the superimposed image.

11. A computer executable program code on a computer readable storage medium for projecting a superimposed image onto a target display surface under observation of at least one camera, the program code comprising:

(a) a first program sub-code for determining a projective relationship between each of a plurality of projectors and the target display surface;

(b) a second program sub-code for estimating a component image for each said projector, said second program sub-code comprising instructions for;

(1) using said projective relationship, for each of said component images determine a plurality of sub-sampled, regionally shifted images represented in the frequency domain; and

(2) composing each said component image using a respective plurality of said sub-sampled, regionally shifted images; and

(c) a third program sub-code for minimizing the difference between a sum of said component images and a frequency domain representation of a target image to produce a second component image for each said projector.

12. The program code of claim 11 wherein said frequency domain representation of said target image as _{FT[u, v]}, and said third program sub-code comprises instructions for:

(a) minimizing the expression;

${<br><br>F_T<br><br>\left[u,v\right]-\left(\underset{i=0}{\overset{k}{\sum <br><br>}}<br><br>F_{\mathrm{\Delta <br><br>}}^i<br><br>\left[u,v\right]\right)<br><br>}^2$

wherein k represents the total number of said plurality of projectors, said component image for each said projector is represented by

$F_{\mathrm{\Delta <br><br>}}^p<br><br>\left[u,v\right]=\underset{r=0}{\overset{N}{\sum <br><br>}}<br><br>F_{\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>r}<br><br>\left[u,v\right],$

and said sub-sampled, regionally shifted images represented in the frequency domain are represented by <sub>FΔ

r[u, v]</sub>, to identify a second set of frequency domain coefficients for use in producing a frequency domain representation of said second component image for a respective one of said projectors; and

(b) taking the inverse Fourier transform of said frequency domain representation of said second component image for said respective one of said projectors.

13. A computer executable program code on a computer readable storage medium for projecting a superimposed image onto a target display surface under observation of at least one camera, the program code comprising:

(a) a first program sub-code for determining a projective relationship between each of a plurality of projectors and the target display surface;

(b) a second program sub-code for estimating a component image for each said projector, said second program sub-code comprising instructions for;

(1) using said projective relationship, for each of said component images determine a plurality of sub-sampled, regionally shifted images represented in the frequency domain; and

(2) composing each said component image using a respective plurality of said sub-sampled, regionally shifted images; and

(c) a third program sub-code for performing an optimization using said component images and a frequency domain representation of a target image to produce a second component image for each said projector.

14. The program code of claim 13 wherein:

(a) said third program sub-code comprises instructions for minimizing the difference between a sum of said component images and said frequency domain representation of said target image to produce said second component image for each said projector; and

(b) said first program sub-code comprises instructions for, while observing the target display surface with the camera, c, recovering a homography between each projector, pi, and the camera, _Hpi^b, and using said homography, _Hpi^c, determining said projective relationship, _Hpi^b, wherein b represents a reference frame for the target display surface.