Method and apparatus for displaying images

• US 20040070565A1
• Filed: 10/02/2003
• Published: 04/15/2004
• Est. Priority Date: 12/05/2001
• Status: Abandoned Application

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Nayar, Shree K, Boult, Terrance E., Belhumeur, Peter
• Application Number:
  US10/416,069
• Publication Number:
  US 20040070565A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G09G 2320/0233 Improving the luminance or brightness uniformity across the screen
• CPC Class Codes: G09G 2320/0626 for control of overall brightness
• CPC Class Codes: G09G 2320/0686 with two or more screen areas displaying information with different brightness or colours
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

An image-displaying method and apparatus adds, or compensates for, effects associated with environmental lighting shining on the display region, and/or imperfections in the display system hardware or display surface. By detecting the environmental illumination, the system can render an image which simulates 2-d or 3-d content (i.e., objects) as if the content were actually illuminated by the environmental lighting. Information regarding the environmental lighting can also be used to cancel out spurious bright spots caused by environmental lighting patterns shining on the display region. In addition, the image displayed in the display region can be monitored for accuracy, and can be adjusted to correct for errors caused by, e.g., spurious bright spots, imperfections in the display system characteristics, and/or imperfections in or on the surface of the display region.

Claims

1. A method for displaying images, comprising:

exposing a display region to first external light comprising a first light ray from a first light source;

receiving first information comprising an approximation of a first characteristic of the first light ray, the first characteristic comprising at least one of a first location of the first light ray, a first direction of the first light ray, a first brightness value of the first light ray, and a first color value of the first light ray;

receiving second information comprising at least one characteristic of an object, the at least one characteristic of the object comprising at least one of a geometrical characteristic and a reflectance characteristic;

using the first and second information to generate a first image of the object, the first image approximating a first view of the object illuminated by the first external light; and

displaying the first image in the display region.

2. A method according to claim 1, further comprising detecting at least one of the first light ray and a second light ray coming from the first external light source for generating the first information.

3. A method according to claim 2, wherein the first characteristic comprises the first direction of the first light ray, and the method further comprises:

generating an image of the first external light; and

using the image of the first external light to generate the first information.

4. A method according to claim 3, wherein the first information further comprises an approximation of a second characteristic comprising at least one of the first brightness value of the first light ray and the first color value of the first light ray, the method further comprising:

receiving third information comprising an approximation of the second characteristic; and

using the third information to generate the first image of the object.

5. A method according to claim 1, further comprising:

detecting a second light ray from the first light source for generating third information comprising an approximation of a second characteristic of the second light ray, the second characteristic comprising at least one of a second brightness value of the second light ray and a second color value of the second light ray;

detecting a third light ray from the first light source for generating fourth information comprising an approximation of a third characteristic of the third light ray, the third characteristic comprising at least one of a third brightness value of the third light ray and a third color value of the third light ray; and

using the third and fourth information to determine the first information.

6. A method according to claim 1, further comprising:

reflecting, by at least one reflective element, at least one of the first light ray and a second light ray from the first external light source, for generating a third light ray; and

detecting the third light ray for generating the first information.

7. A method according to claim 1, further comprising:

detecting a first light ray bundle, including a first chief ray, from the first light source for determining a first direction of the first chief ray;

detecting a second light ray bundle, including a second chief ray, from the first light source for determining a second direction of the second chief ray;

using the first and second directions to determine a three-dimensional location of the first light source; and

using the three-dimensional location to generate the first information.

8. A method according to claim 1, wherein the step of using the first and second information comprises using the first information to generate a model light source pattern for approximating the first external light source, the model light source pattern comprising one of:

a point intensity pattern; and

a distributed intensity pattern having a non-zero, approximately uniform intensity value within a light source region having a selected geometric shape, the distributed intensity pattern having an approximately zero intensity value outside the light source region.

9. A method according to claim 1, wherein the first light ray is incident upon a first portion of the display region at a first time, the method further comprising:

exposing the display region at a second time to second external light comprising a second light ray from one of the first light source and a second light source, the first and the second light rays being incident upon the first portion of the display region;

receiving third information comprising an approximation of a second characteristic of the second light ray, the second characteristic comprising at least one of a second location of the second light ray, a second direction of the second light ray, a second brightness value of the second light ray, and a second color value of the second light ray;

using the second and third information to generate a second image of the object, the second image approximating a second view of the object illuminated by the second light; and

displaying the second image in the display region at approximately the second time.

10. A method according to claim 1, further comprising receiving third information comprising an approximation of a location of a viewer viewing the first image, the at least one characteristic of the object comprising a directional reflectance characteristic of the object, and the step of using the first and second information including using the third information to generate the first image.

11. A method for displaying images, comprising:

providing to a display device, a first signal representing a characteristic of at least a portion of a first image to be displayed, the characteristic of the at least a portion of the first image comprising at least one of a first brightness value and a first color value;

using the first signal to cause the display device to display at least a portion of a second image at a first time and in a first portion of a display region, the at least a portion of the second image comprising an approximation of the at least a portion of the first image;

detecting, at approximately the first time, a first light signal from the first portion of the display region for determining a characteristic of the first light signal, the characteristic of the first light signal comprising at least one of a second brightness value and a second color value;

determining a first difference between the characteristic of at least a portion of the first image and the characteristic of the first light signal; and

using the first difference to determine a first adjustment in the display of a portion of an image in the first portion of the display region of the display device.

12. A method according to claim 11, further comprising:

receiving a second signal representing a third characteristic of at least a portion of a third image, the third characteristic comprising at least one of a third brightness value and a third color value;

adjusting the second signal to obtain the first adjustment in the display of a portion of an image in the first portion of the display region for generating a third signal; and

using the third signal to cause the display device to display at least a portion of a fourth image at a second time and in the first portion of the display region, the second time being after the first time.

13. A method according to claim 12, wherein the at least a portion of the fourth image has a fourth characteristic comprising at least one of a fourth brightness value and a fourth color value, the step of using the first difference comprising using a lookup table to determine an approximate amount of change of the fourth characteristic associated with the step of adjusting the second signal to obtain the first adjustment in the display of a portion of an image in the first portion of the display region.

14. A method according to claim 12, further comprising:

adjusting the second signal to obtain a global adjustment in the display of an image in the display region, for generating the third signal, the global adjustment being sufficiently large to ensure that the third signal represents a non-negative brightness value; and

adjusting a brightness value of at least a portion of a fifth image to obtain the global adjustment in the display of an image in the display region, the at least a portion of the fifth image being displayed at the second time and in a second portion of the display region.

15. A method according to claim 11, wherein the first image portion comprises a first pixel, the second image portion comprising a second pixel, and the method further comprising:

providing to the display device a third signal representing a third characteristic of a third pixel, the third characteristic comprising at least one of a third brightness value and a third color value;

using the third signal to cause the display device to display a fourth pixel in a second portion of the display region, the fourth pixel comprising an approximation of the third pixel;

detecting, during the step of using the third signal to display the fourth pixel, a second light signal from the second portion of the display region for determining a fourth characteristic of the second light signal, the fourth characteristic comprising at least one of a fourth brightness value and a fourth color value;

determining a second difference between the third and fourth characteristics; and

using the second difference to determine a second adjustment in the display of a portion of an image in the second portion of the display region.

16. A method for displaying images, comprising:

receiving a first signal representing a first characteristic of at least a portion of a first image, the first characteristic comprising at least one of a first brightness value and a first color value;

exposing a first portion of a display region to first external light comprising a first light ray from a first light source;

receiving first information comprising an approximation of a second characteristic of the first light ray, the second characteristic comprising at least one of a location of the first light ray, a direction of the first light ray, a second brightness value of the first light ray, and a second color value of the first light ray;

using the first information to determine a third characteristic of a first approximately non-directionally reflected light signal from the first portion of the display region, the third characteristic comprising at least one of a third brightness value and a third color value, and the first approximately non-directionally reflected light signal being caused by the first light ray;

using the third characteristic to determine an adjustment of the first signal;

adjusting the first signal by the adjustment for generating an adjusted signal; and

using the adjusted signal to cause the display of at least a portion of a second image in the first portion of the display region.

17. A method according to claim 16, further comprising detecting at least one of the first light ray and a second light ray from the first light source for generating the first information.

18. A method according to claim 17, wherein the first characteristic comprises the direction of the first light ray, and the method further comprising:

generating a light source image; and

using the light source image to generate the first information.

19. A method according to claim 16, further comprising:

detecting a first incident light signal from the first light source for generating second information comprising at least a fourth characteristic of the first incident light signal, the fourth characteristic comprising at least one of a fourth brightness of the first incident light signal and a fourth color of the first incident light signal;

detecting a second incident light signal from the first light source for generating third information regarding a fifth characteristic of the second incident light signal, the fifth characteristic comprising at least one of a fifth brightness of the second incident light signal and a fifth color of the second incident light signal; and

using the second and third information to determine the first information.

20. A method according to claim 16, further comprising:

reflecting, by at least one reflective element, at least one of the first light and second light from the at least one light source, for generating third light; and

detecting the third light for generating the first information.

21. A method according claim 16, further comprising:

detecting a first light ray bundle having a first chief ray from the first light source for determining a first direction of the first chief ray;

detecting a second light ray bundle having a second chief ray from the first light source, for determining a second direction of the second chief ray;

using the first and second directions to determine a three-dimensional location of the first light source; and

using the three-dimensional location to generate the first information.

22. A method according to claim 16, wherein the step of using the first information comprises using the first information to generate a model light source pattern for approximating the at least one light source, the model light source pattern comprising one of:

a point intensity pattern; and

a distributed intensity pattern having a non-zero, approximately uniform intensity value within a light source region having a selected geometric shape, the distributed intensity pattern having an approximately zero intensity value outside the light source region.

23. An apparatus for displaying images, comprising:

a display region exposed to first external light comprising a first light ray from a first light source;

a first processor for receiving first information comprising an approximation of a first characteristic of the first light ray, the first characteristic comprising at least one of a first location of the first light ray, a first direction of the first light ray, a first brightness value of the first light ray, and a first color value of the first light ray;

a second processor for receiving second information comprising at least one characteristic of an object, the at least one characteristic of the object comprising at least one of a geometrical characteristic and a reflectance characteristic;

a third processor for using the first and second information to generate a first image of the object, the first image approximating a first view of the object illuminated by the first external light; and

a display device for displaying the first image in the display region.

24. An apparatus according to claim 23, further comprising at least one detector for detecting at least one of the first light ray and a second light ray coming from the first external light source for generating the first information.

25. An apparatus according to claim 24, wherein the first characteristic comprises the first direction of the first light ray, the at least one detector comprising an imager for generating an image of the first external light, and the apparatus further comprising a fourth processor for using the image of the first external light to generate the first information.

26. An apparatus according to claim 25, wherein the first information further comprises an approximation of a second characteristic comprising at least one of the first brightness value of the first light ray and the first color value of the first light ray, the apparatus further comprising:

a fourth processor for receiving third information comprising an approximation of the second characteristic; and

a fifth processor for using the third information to generate the first image of the object.

27. An apparatus according to claim 23, further comprising:

a first detector for detecting a second light ray from the first light source for generating third information comprising an approximation of a second characteristic of the second light signal, the second characteristic comprising at least one of a second brightness value of the second light ray and a second color value of the second light ray;

a second detector for detecting a third light ray from the first light source for generating fourth information comprising an approximation of a third characteristic of the third light ray, the third characteristic comprising at least one of a third brightness value of the third light ray and a third color value of the third light ray; and

a fourth processor for using the third and fourth information to determine the first information.

28. An apparatus according to claim 23, further comprising:

at least one reflective element for reflecting at least one of the first light ray and a second light ray from the first external light source for generating a third light ray; and

an imager for detecting the third light ray for generating the first information.

29. An apparatus according to claim 23, further comprising:

a first detector for detecting a first light ray bundle, including a first chief ray, from the first light source for determining a first direction of the first chief ray;

a second detector for detecting a second light ray bundle, including a second chief ray, from the first light source for determining a second direction of the second chief ray;

a fourth processor for using the first and second directions to determine a three-dimensional location of the first light source; and

a fifth processor for using the three-dimensional location to generate the first information.

30. An apparatus according to claim 23, wherein the third processor comprises a fourth processor for using the first information to generate a model light source pattern for approximating the first external light source, the model light source pattern comprising one of:

a point intensity pattern; and

a distributed intensity pattern having a non-zero, approximately uniform intensity value within a light source region having a selected geometric shape, the distributed intensity pattern having an approximately zero intensity value outside the light source region.

31. An apparatus according to claim 23, wherein the display region comprises a first display region portion, the first light ray being incident upon the first display region portion at a first time, the display region being further exposed, at a second time, to second external light comprising a second light ray from one of the first light source and a second light source, the second light ray being incident upon the first portion of the display region, and the apparatus further comprising:

a fourth processor for receiving third information comprising an approximation of a second characteristic of the second light ray, the second characteristic comprising at least one of a second location of the second light ray, a second direction of the second light ray, a second brightness value of the second light ray, and a second color value of the second light ray;

a fifth processor for using the second and third information to generate a second image of the object, the second image approximating a second view of the object illuminated by second light comprising the second light ray; and

a sixth processor for controlling the display device to display the second image in the display region at approximately the second time.

32. An apparatus according to claim 23, further comprising a fourth processor for receiving third information comprising an approximation of a location of a viewer viewing the first image, the at least one characteristic of the object comprising a directional reflectance characteristic of the object, and the third processor comprising a sixth processor for using the third information to generate the first image.

33. An apparatus for displaying images, comprising:

a display device for using a first signal to display at least a portion of a second image at a first time and in a first portion of a display region, the first signal representing a characteristic of at least a portion of a first image to be displayed, the characteristic of the at least a portion of the first image comprising at least one of a first brightness value and a first color value, and the at least a portion of the second image comprising an approximation of the at least a portion of the first image;

a detector for detecting, at approximately the first time, a first light signal from the first portion of the display region for determining a characteristic of the first light signal, the characteristic of the first light signal comprising at least one of a second brightness value and a second color value;

a first processor for determining a first difference between the characteristic of at least a portion of the first image and the characteristic of the first light signal; and

a second processor for using the first difference to determine a first in the display of a portion of an image in the first portion of the display region of the display device.

34. An apparatus according to claim 33, further comprising:

a third processor for receiving a second signal representing a third characteristic of at least a portion of a third image, the third characteristic comprising at least one of a third brightness value and a third color value;

a fourth processor for adjusting the second signal to obtain the first adjustment in the display of a portion of an image in the first portion of the display region, for generating a third signal; and

a fifth processor for controlling the display device to use the third signal to display at least a portion of a fourth image at a second time and in the first portion of the display region, the second time being after the first time.

35. An apparatus according to claim 34, wherein the at least a portion of the fourth image has a fourth characteristic comprising at least one of a fourth brightness value and a fourth color value, the second processor comprising a sixth processor for using a lookup table to determine an approximate amount of change of the fourth characteristic associated with adjusting the second signal to obtain the first amount of adjustment in the display of a portion of an image in the first portion of the display region.

36. An apparatus according to claim 34, further comprising:

a sixth processor for adjusting the second signal to obtain a global adjustment in the display of an image in the display region, for generating the third signal, the global adjustment being sufficiently large to ensure that the third signal represents a non-negative brightness value;

a seventh processor for adjusting a brightness value of at least a portion of a fifth image to obtain the global adjustment in the display of an image in the display region; and

an eighth processor for controlling the display device to display the at least a portion of the fifth image at the second time and in a second portion of the display region.

37. An apparatus according to claim 33, wherein the first image portion comprises a first pixel, the second image portion comprising a second pixel, the display device receiving a third signal representing a third characteristic of a third pixel, the third characteristic comprising at least one of a third brightness value and a third color value, the apparatus further comprising:

a third processor for using the third signal to control the display device to display a fourth pixel in a second portion of the display region, the fourth pixel comprising an approximation of the third pixel;

a fourth processor for controlling the detector to detect, during the step of using the third signal to display the fourth pixel, a second light signal from the second portion of the display region for determining a fourth characteristic of the second light signal, the fourth characteristic comprising at least one of a fourth brightness value and a fourth color value;

a fifth processor for determining a second difference between the third and fourth characteristics; and

a sixth processor for using the second difference to determine a second adjustment in the display of a portion of an image in the second portion of the display region.

38. An apparatus for displaying images, comprising:

a first processor for receiving a first signal representing a first characteristic of at least a portion of a first image, the first characteristic comprising at least one of a first brightness value and a first color value;

a display region having a first display region portion exposed to first external light comprising a first light ray from a first light source;

a second processor for receiving first information comprising an approximation of a second characteristic of the first light ray, the second characteristic comprising at least one of a location of the first light ray, a direction of the first light ray, a second brightness value of the first light ray, and a second color value of the first light ray;

a third processor for using the first information to determine a third characteristic of a first approximately non-directionally reflected light signal from the first display region portion, the third characteristic comprising at least one of a third brightness value and a third color value, and the first approximately non-directionally reflected light signal being caused by the first light ray;

a fourth processor for using the third characteristic to determine an adjustment of the first signal;

a fifth processor for adjusting the first signal by the adjustment for generating an adjusted signal; and

a display device for using the adjusted signal to display at least a portion of a second image in the first display region portion.

39. An apparatus according to claim 38, further comprising at least one detector for detecting at least one of the first light ray and a second light ray from the first light source for generating the first information.

40. An apparatus according to claim 39, wherein the first characteristic comprises the direction of the first light ray, the at least one detector comprising an imager for generating a light source image, and the apparatus further comprising a sixth processor for using the light source image to generate the first information.

41. An apparatus according to claim 38, further comprising:

a first detector for detecting a first incident light signal from the first light source for generating second information comprising at least a fourth characteristic of the first incident light signal, the fourth characteristic comprising at least one of a fourth brightness of the first incident light signal and a fourth color of the first incident light signal;

a second detector for detecting a second incident light signal from the first light source for generating third information regarding a fifth characteristic of the second incident light signal, the fifth characteristic comprising at least one of a fifth brightness of the second incident light signal and a fifth color of the second incident light signal; and

a sixth processor for using the second and third information to determine the first information.

42. An apparatus according to claim 38, further comprising:

at least one reflective element for reflecting at least one of the first light and second light from the at least one light source, for generating third light; and

an imager for detecting the third light, for generating the first information.

43. An apparatus according to claim 38, further comprising:

a first detector for detecting a first light ray bundle having a first chief ray from the first light source for determining a first direction of the first chief ray;

a second detector for detecting a second light ray bundle having a second chief ray from the first light source, for determining a second direction of the second chief ray;

a sixth processor for using the first and second directions to determine a three-dimensional location of the first light source; and

a seventh processor for using the three-dimensional location to generate the first information.

44. An apparatus according to claim 38, wherein the third processor comprises a sixth processor for using the first information to generate a model light source pattern for approximating the at least one light source, the model light source pattern comprising one of:

a point intensity pattern; and

a distributed intensity pattern having a non-zero, approximately uniform intensity value within a light source region having a selected geometric shape, the distributed intensity pattern having an approximately zero intensity value outside the light source region.