Dynamic shadow removal from front projection displays

• US 20030128337A1
• Filed: 12/09/2002
• Published: 07/10/2003
• Est. Priority Date: 12/07/2001
• Status: Active Grant

• Current Assignee: Keach Donald G PHD Director of Intellectual Property For and On Behalf of University of Kentucky Research Foundation
• Original Assignee: Keach Donald G PHD Director of Intellectual Property For and On Behalf of University of Kentucky Research Foundation
• Inventors: Jaynes, Christopher O., Webb, Stephen B., Steele, Robert M.
• Granted Patent: US 7,133,083 B2
• Time in Patent Office: Days
• US Class Current: 353/30
• CPC Class Codes: G03B 21/26 Projecting separately subsidiary matter simultaneously with main image

Abstract

A technique and system for detecting a radiometric variation/artifacts of a front-projected dynamic display region under observation by at least one camera. The display is comprised of one or more images projected from one or more of a plurality of projectors; the system is preferably calibrated by using a projective relationship. A predicted image of the display region by the camera is constructed using frame-buffer information from each projector contributing to the display, which has been geometrically transformed for the camera and its relative image intensity adjusted. A detectable difference between a predicted image and the display region under observation causes corrective adjustment of the image being projected from at least one projector. The corrective adjustment may be achieved by way of pixel-wise approach (an alpha-mask is constructed from delta pixels/images), or bounding region approach (difference/bounding region is sized to include the area of the display affected by the radiometric variation). Also: a technique, or method, for detecting a radiometric variation of a display region under observation, as well as associated computer executable program code on a computer readable storage medium, therefor.

Claims

1. A system for detecting a radiometric variation of a front-projected display region under observation by at least one camera at a first location, the system comprising:

the display comprising at least one image projected from each of a plurality of projectors; and

at least one processing unit for (a) constructing a predicted image of the display region by the camera using framebuffer information from each of said projectors that has been geometrically transformed for the camera, and for which image intensity has been adjusted, and (b) comparing said predicted image with the display region under observation for the detecting.

2. The system of claim 1 wherein a detectable difference between said predicted image and the display region under observation causes a corrective adjustment of said image being projected from at least one of said plurality of projectors.

3. The system of claim 2 wherein said detectable difference comprises a first and second set of delta image pixels which are, thereafter, mapped to a framebuffer of said at least one projector for said corrective adjustment;

said at least one projector has an unobstructed projection-path to the radiometric variation of the display; and

said corrective adjustment comprises blending an alpha-mask constructed from said first and second sets of delta image pixels.

4. The system of claim 3 wherein said geometrically transformed information comprises pixel information for said image projected by a respective one of said projectors, that has been transformed into a frame of the camera using a projective relationship by identifying a projector-to-camera homography for each said projector and a projector-to-projector homography for said projectors;

and said mapping to said framebuffer of said projector comprises using an inverse of said projector-to-camera homography.

5. The system of claim 2 wherein said at least one projector has an unobstructed projection-path to a bounding region sized to encompass the radiometric variation of the display;

said corrective adjustment comprises projecting, from said projector with said unobstructed projection-path, image information for that portion of the display within said bounding region, said image information being projected for a time period after said difference is detectable.

6. The system of claim 1 wherein said plurality comprises a first and second projector;

and further comprising a third projector having an unobstructed projection-path to a bounding region sized to encompass the radiometric variation of the display; and

wherein a detectable difference between said predicted image and the display region under observation causes a corrective adjustment, said corrective adjustment to comprise projecting, from said third projector, image information for that portion of the display within said bounding region, said image information being projected for a time period after said difference is detectable; and

during said time period, each of said plurality of projectors affected by the radiometric variation projects no image within said bounding region.

7. The system of claim 1 wherein said geometrically transformed information comprises pixel information for said image projected by a respective one of said projectors, that has been transformed into a frame of the camera using a projective relationship;

and said adjustment of said image intensity comprises constructing a color transfer function, fc(x), for at least one color channel, c, to provide a mapping of said intensity of said pixel information for said image projected by said respective projector into said frame of the camera.

8. The system of claim 7 wherein said color transfer function is of the form:

9. A system for detecting a radiometric variation of a front-projected display region under observation by at least one camera at a first location, the system comprising:

the display comprising at least one image projected from at least one of a plurality of projectors;

at least one processing unit for (a) constructing a predicted image of the display region by the camera using framebuffer information from said at least one projector that has been geometrically transformed for the camera, and for which image intensity has been adjusted, and (b) comparing said predicted image with the display region under observation for the detecting; and

wherein a detectable difference between said predicted image and the display region under observation causes a corrective adjustment comprising projecting image information from another of said plurality of projectors for a time period after said difference is detectable.

10. A method for detecting a radiometric variation of a front-projected display region under observation by at least one camera at a first location, the method comprising the steps of:

providing the display by projecting at least one image from each of a plurality of projectors;

constructing a predicted image of the display region using framebuffer information from each said projector, said step of constructing to comprise geometrically transforming each said framebuffer information to the camera, and adjusting image intensity of said geometrically transformed framebuffer information; and

comparing said predicted image with the display region under observation for the detecting.

11. The method of claim 10 wherein, upon performing said step of comparing, a difference detected between said predicted image and the display region under observation causes a corrective adjustment of said image being projected from at least one of said plurality of projectors.

12. The method of claim 11 further comprising, in the event said difference is detected, the step of mapping a first and second set of delta image pixels to a framebuffer of said at least one projector for said corrective adjustment;

and wherein said at least one projector has an unobstructed projection-path to the radiometric variation of the display; and

said corrective adjustment comprises blending an alpha-mask constructed from said first and second sets of delta image pixels with an alpha channel of said image projected from said at least one projector.

13. The method of claim 12 further comprising the steps of identifying a projector-to-camera homography for each said projector and a projector-to-projector homography for said projectors for use in said step of geometrically transforming;

and wherein;

each said framebuffer information comprises pixel information for said image projected by a respective one of said projectors;

said step of geometrically transforming further comprises using a projective relationship comprising said homographies to transform said pixel information for said image projected by said respective projector, into a frame of the camera; and

said step of mapping said first and second set of delta image pixels to said framebuffer of said projector comprises using an inverse of said projector-to-camera homography.

14. The method of claim 11 wherein said at least one projector has an unobstructed projection-path to a bounding region sized to encompass the radiometric variation of the display;

said corrective adjustment comprises projecting, from said projector with said unobstructed projection-path, image information for that portion of the display within said bounding region, said image information being projected for a time period after said difference is detectable; and

during said time period, each of said plurality of projectors affected by the radiometric variation projects no image within said bounding region.

15. The method of claim 10 wherein said plurality comprises a first and second projector;

and further comprising a third projector having an unobstructed projection-path to a bounding region sized to encompass the radiometric variation of the display; and

wherein, upon performing said step of comparing, a difference detected between said predicted image and the display region under observation causes a corrective adjustment, said corrective adjustment to comprise projecting, from said third projector, image information for that portion of the display within said bounding region, said image information being projected for a time period after said difference is detectable.

16. The method of claim 10 wherein:

each said framebuffer information comprises pixel information for said image projected by a respective one of said projectors;

said step of geometrically transforming further comprises using a projective relationship to transform said pixel information for said image projected by said respective projector, into a frame of the camera; and

said step of adjusting image intensity comprises constructing a color transfer function, fc(x), for at least one color channel, c, to provide a mapping of said intensity of said pixel information for said image projected by said respective projector into said frame of the camera.

17. The system of claim 16 wherein said color transfer function is of the form:

18. A method for detecting a radiometric variation of a front-projected display region under observation by at least one camera at a first location, the method comprising the steps of:

providing the display by projecting at least one image from at least one of a plurality of projectors;

constructing a predicted image of the display region using framebuffer information from said at least one projector, said step of constructing to comprise geometrically transforming said framebuffer information to the camera, and adjusting image intensity of said geometrically transformed framebuffer information; and

upon comparing said predicted image with the display region under observation, a difference detected therebetween causes a corrective adjustment comprising projecting image information from another of said plurality of projectors for a time period after said difference is detectable.

19. A computer executable program code on a computer readable storage medium for detecting a radiometric variation of a front-projected display region under observation by at least one camera at a first location, the program code comprising:

a first program sub-code for providing the display by projecting at least one image from each of a plurality of projectors;

a second program sub-code for constructing a predicted image of the display region using framebuffer information from each said projector, said second sub-code comprising instructions for geometrically transforming each said framebuffer information to the camera, and instructions for adjusting image intensity of said geometrically transformed framebuffer information; and

a third program sub-code for comparing said predicted image with the display region under observation for the detecting.

20. The program code of claim 19 wherein:

each said framebuffer information comprises pixel information for said image projected by a respective one of said projectors; and

said instructions for geometrically transforming comprise instructions for using a projective relationship to transform said pixel information for said image projected by said respective projector, into a frame of the camera; and

further comprising a fourth sub-code for causing a corrective adjustment of said image being projected from at least one of said plurality of projectors in the event a difference is detected between said predicted image and the display region under observation.

21. A computer executable program code on a computer readable storage medium for detecting a radiometric variation of a front-projected display region under observation by at least one camera at a first location, the program code comprising:

a first program sub-code for providing the display by projecting at least one image from at least one of a plurality of projectors;

second program sub-code for constructing a predicted image of the display region using framebuffer information from said at least one projector, said second sub-code comprising instructions for geometrically transforming said framebuffer information to the camera, and instructions for adjusting image intensity of said geometrically transformed framebuffer information; and

a third program sub-code for comparing said predicted image with the display region under observation, whereupon a difference detected therebetween causes a corrective adjustment comprising projecting image information from another of said plurality of projectors for a time period during which said difference is detectable.