Methods and systems for compensating an image projected onto a surface having spatially varying photometric properties

• US 20050134599A1
• Filed: 07/02/2004
• Published: 06/23/2005
• Est. Priority Date: 07/02/2003
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Grossberg, Michael, Nayar, Shree, Belhumeur, Peter, Peri, Harish
• Granted Patent: US 7,663,640 B2
• Time in Patent Office: Days
• US Class Current: 345/589
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image

Abstract

Methods and systems are provided for displaying images onto an arbitrary surface, using a projector, such that the quality of the images is preserved despite surface imperfections or color variations. Methods and systems are also provided for controlling the appearance of a projection surface. Various embodiments use a detailed radiometric model and a calibration method to determine the pixel values required to be projected by a projector in order for a camera to observe a desired image. Other embodiments use a compensation algorithm that uses a feedback approach to provide the desired image compensation. Geometric mapping may be used to establish a correspondence between points in the images to be displayed by the projector and the corresponding points in the images that are captured by the camera.

Claims

1. A method for projecting a display image onto a surface, wherein the surface has spatially varying photometric properties, the method comprising the steps of:

performing radiometric calibration on the surface based on a radiometric model;

compensating the display image to form a compensated display image based on the radiometric calibration so that a measured image has desired characteristics; and

projecting the compensated display image onto the surface.

2. The method of claim 1, wherein the performing radiometric calibration step comprises calculating entries for a color mixing matrix V.

3. The method of claim 1, wherein the calculating step comprises:

projecting display images having different colors;

capturing the display images as measured images; and

calculating the entries for the color mixing matrix V based on the measured images.

4. The method of claim 1, wherein the performing radiometric calibration step comprises computing the non-linear responses of a projector used to project the compensated display image and a camera used to capture the measured image.

5. The method of claim 4, wherein the computing step comprises:

projecting display images having increasing gray intensity;

capturing the display images as measured images; and

multiplying color values in the measured images by the inverse of a color mixing matrix V.

6. The method of claim 1, wherein the radiometric model describes a relationship between the brightness in display images and the brightness in measured images.

7. The method of claim 1, wherein the radiometric model comprises a color mixing matrix V.

8. The method of claim 1, further comprising the step of determining a geometric mapping between a projector used to project the compensated display image and a camera used to capture the measured image.

9. The method of claim 8, wherein the determining step comprises using a piecewise polynomial model to establish a correspondence between points in the compensated display image and points in the measured image.

10. The method of claim 8, wherein the determining step comprises:

creating a plurality of calibration images

projecting the plurality of calibration images;

capturing the calibration images as measured calibration images; and

mapping points in the calibration images to points in the measured calibration images.

11. The method of claim 8, wherein the geometric mapping is predetermined at the time of manufacture of the projector and the camera.

12. The method of claim 11, wherein the optics of the projector and the camera are coaxial.

13. The method of claim 1, further comprising the steps of:

capturing the compensated display image as a second measured image;

comparing the second measured image with the display image;

compensating the compensated display image to form a second compensated display image based on the comparison between the second measured image and the display image; and

projecting the second compensated display image.

14. The method of claim 13, further comprising repeatedly performing the capturing step, the comparing step, the compensating step, and the projecting the second compensated display image step until the second measured image is within a tolerance of the display image.

15. The method of claim 1, wherein the display image is a still image.

16. The method of claim 15, wherein the display image is an advertisement.

17. The method of claim 1, wherein the display image is a video frame.

18. The method of claim 1, wherein the projected compensated display image is used to change the appearance of the surface.

27. The method of claim 1, wherein the projected compensated display image is used to change the appearance of the surface.

19. A method for projecting a display image onto a surface having spatially varying photometric properties comprising the steps of:

projecting a projected image based on the display image onto the surface;

capturing the projected image as a measured image;

comparing the measured image with the display image;

compensating the projected image based on the comparison between the measured image and the display image; and

projecting the compensated projected image onto the surface.

20. The method of claim 19, further comprising repeatedly performing the capturing step, the comparing step, the compensating step, and the projecting the compensated projected image step until the measured image is within a tolerance of the display image.

21. The method of claim 19, further comprising the step of determining a geometric mapping between the projector and the camera.

22. The method of claim 21, wherein the geometric mapping is predetermined at the time of manufacture of the projector and the camera.

23. The method of claim 22, wherein the optics of the projector and the camera are coaxial.

24. The method of claim 19, wherein the display image is a still image.

25. The method of claim 24, wherein the display image is an advertisement.

26. The method of claim 19, wherein the display image is a video frame.

28. A method for projecting a display image onto a surface, wherein the surface has spatially varying photometric properties, the method comprising the steps of:

detecting the spatially varying photometric properties of the surface;

compensating the display image based on the detected spatially varying photometric properties; and

projecting the compensated display image onto the surface.

29. A system for projecting a display image onto a surface having spatially varying photometric properties, the system comprising:

a projector that projects images onto the surface;

a camera that captures the projected images as measured images; and

a processor coupled to the projector and the camera that performs radiometric calibration on a radiometric model based on projected images and measure images, that compensates the display image to form a compensated display image based on the radiometric calibration so that a measured image has desired characteristics, and that provides the compensated display image to the projector for projection onto the surface.

30. The system of claim 29, wherein the radiometric model describes a relationship between the brightness in display images and the brightness in measured images.

31. The system of claim 29, wherein the radiometric model comprises a color mixing matrix V.

32. The system of claim 31, wherein the processor, in performing radiometric calibration, calculates entries for the color mixing matrix V.

33. The system of claim 32, wherein the processor, in calculating, causes display images having different colors to be projected, causes the display images to be captured as measured images, and calculates the entries for the color mixing matrix V based on the measured images.

34. The system of claim 29, wherein the processor, in performing radiometric calibration, computes the non-linear responses of the projector and the camera.

35. The system of claim 34, wherein the processor, in computing, causes display images having increasing gray intensity to be projected, causes the display images to be captured as measured images, and multiplies color values in the measured images by the inverse of a color mixing matrix V.

36. The system of claim 29, wherein the processor also determines a geometric mapping between the projector and the camera.

37. The system of claim 36, wherein the processor, in determining, uses a piecewise polynomial model to establish a correspondence between points in the compensated display image and points in the measured image.

38. The system of claim 36, wherein the processor, in determining, creates a plurality of calibration images, causes the plurality of calibration images to be projected, causes the calibration images to be captured as measured calibration images, and maps points in the calibration images to points in the measured calibration images.

39. The system of claim 29, wherein a geometric mapping between the projector and the camera is predetermined at the time of manufacture of the projector and the camera.

40. The method of claim 39, wherein the optics of the projector and the camera are coaxial.

41. The system of claim 29, wherein the processor also causes the compensated display image to be captured as a second measured image, compares the second measured image with the display image, compensates the compensated display image to form a second compensated display image based on the comparison between the second measured image and the display image, and causes the second compensated display image to be projected.

42. The system of claim 41, wherein the processor also repeatedly causes the compensated display image to be captured, compares the second measured image with the display image, compensates the compensated display image, and causes the second compensated display image to be projected until the second measured image is within a tolerance of the display image.

43. A system for projecting a display image onto a surface having spatially varying photometric properties, the system comprising:

a projector that projects a projected image based on the display image onto the surface;

a camera that captures the projected image as a measured image;

a processor coupled to the projector and the camera that compares the measured image with the display image, compensates the projected image based on the comparison between the measured image and the display image, and causes the compensated projected image to be projected onto the surface.

44. The system of claim 43, wherein the processor also repeatedly causes the compensated display image to be captured, compares the second measured image with the display image, compensates the compensated display image, and causes the second compensated display image to be projected onto the surface until the second measured image is within a tolerance of the display image.

45. The system of claim 43, wherein the processor also determines a geometric mapping between the projector and the camera.

46. The method of claim 43, wherein a geometric mapping between the projector and the camera is predetermined at the time of manufacture of the projector and the camera.

47. The method of claim 46, wherein the optics of the projector and the camera are coaxial.