Materials and methods for simulating focal shifts in viewers using large depth of focus displays

• US 7,428,001 B2
• Filed: 03/07/2003
• Issued: 09/23/2008
• Est. Priority Date: 03/15/2002
• Status: Expired due to Term

• Current Assignee: University of Washington, Washington University In St Louis
• Original Assignee: University of Washington
• Inventors: Seibel, Eric J., Schowengerdt, Brian T.
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US10/508,753
• Publication Number:
  US 20060232665A1
• Time in Patent Office: 2,027 Days
• Field of Search: 348/47, 348/38, 348/218, 348/46, 348/169, 348/54, 348/39, 348/159, 348/157, 348/57, 348/51, 348/42, 348/44, 348/36, 348/52, 382/190, 382/103, 382/254, 382/255, 382/256, 382/260, 382/268, 382/262, 382/263, 382/264, 382/265, 382/266, 382/267, 382/269, 351/210, 351/209, 351/211, 351/212, 351/245, 351/246, 351/155, 351/57
• US Class Current: 348/51
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/144 for flicker reduction
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2213/002 Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices
• CPC Class Codes: H04N 2213/006 Pseudo-stereoscopic systems, i.e. systems wherein a stereoscopic effect is obtained without sending different images to the viewer's eyes
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A large depth of focus (DOF) display provides an image in which the apparent focus plane is adjusted to track an accommodation (focus) of a viewer'"'"'s eye(s) to more effectively convey depth in the image. A device is employed to repeatedly determine accommodation as a viewer'"'"'s gaze within the image changes. In response, an image that includes an apparent focus plane corresponding to the level of accommodation of the viewer is provided on the large DOF display. Objects that are not at the apparent focus plane are made to appear blurred. The images can be rendered in real-time, or can be pre-rendered and stored in an array. The dimensions of the array can each correspond to a different variable. The images can alternatively be provided by a computer controlled, adjustable focus video camera in real-time.

Claims

1. A method for more accurately conveying depth in an image, comprising the steps of:

(a) displaying an image to a viewer on a large depth of focus display wherein all elements in the image are initially displayed at an optical focus level that is substantially the same for all elements, thereby simulating an unnatural or artificial viewing condition, since elements viewed at different distances from the viewer would naturally appear at different focus levels;

(b) determining an accommodation for an eye of the viewer who is watching the image on the large depth of focus display, as a gaze of the viewer is directed toward an element in the image; and

(c) displaying an image having an apparent focus plane that tracks the accommodation of the viewer by adding blurring to other elements in the image in proportion to their distance in depth from the viewpoint of the viewer, so that as the accommodation of the viewer watching the large depth of focus display changes to focus the eye of the viewer at a different viewing distance, the image that is displayed is changed to more accurately visually convey depth in the image that is displayed, based on the accommodation that was determined and thereby simulating a natural viewing condition.

2. The method of claim 1, wherein the step of determining the accommodation comprises the step of directly measuring the accommodation in at least one eye of the viewer.

3. The method of claim 1, wherein the step of determining the accommodation comprises the steps of:

(a) measuring a vergence of at least one eye of the viewer when watching the large depth of focus display; and

(b) determining the accommodation as a function of the vergence.

4. The method of claim 1, wherein the step of determining the accommodation comprises the steps of:

(a) measuring a gaze direction of the viewer when watching the large depth of focus display; and

(b) anticipating the accommodation of the viewer from the gaze direction.

5. The method of claim 1, further comprising the step of rendering in real-time, each image having an apparent focus plane that tracks the accommodation of the viewer, on the large depth of focus display.

6. The method of claim 5, wherein objects within each image that are farther away from the apparent focus plane in the image are rendered at a lower resolution and contrast, to substantially reduce a computational overhead required for rendering the image on the large depth of focus display.

7. The method of claim 1, further comprising the step of pre-preparing a plurality of images having a range of different apparent focus planes, so that the image having the apparent focus plane that tracks the accommodation of the viewer is selected from the plurality of images that were pre-prepared.

8. The method of claim 7, wherein the plurality of images are arranged in a multi-dimensional array, at least one axis of the multi-dimensional array corresponding to a disposition of the apparent focus plane in the plurality of images.

9. The method of claim 8, wherein each other dimension of the multi-dimensional array corresponds to a different parameter that varies within the plurality of images.

10. The method of claim 9, further comprising the step of enabling the viewer to provide an input that varies a value of a parameter for at least one of the other dimensions, to affect the image provided to the large depth of focus display.

11. The method of claim 10, wherein the parameter comprises one of:

(a) a motion of a camera into a scene comprising the plurality of images;

(b) an orientation of a camera used to image a scene to produce the plurality of images; and

(c) a zoom level of a camera used to produce the plurality of images.

12. The method of claim 8, wherein the image that is displayed by the large depth of focus display is a 2½

-dimensional image comprising visual depth information.

13. The method of claim 7, wherein the plurality of images are pre-prepared by capturing a scene with a camera having a variable focus set at a plurality of different focal planes.

14. The method of claim 1, further comprising the step of producing the image having the apparent focus plane that tracks the accommodation of the viewer by adjusting a focus of a variable focus camera so that the variable focus camera produces said image by imaging a real scene with the focus set at said apparent focus plane.

15. The method of claim 1, further comprising the step of producing successive images having apparent focus planes that track the accommodation of the viewer, at a sufficiently fast image rate to produce a perception of motion of an object within the successive images.

16. The method of claim 1, further comprising the step of producing an image having at least one element that is laterally shifted and having an apparent focus plane that tracks the accommodation of the viewer, so that each eye sees a different image, to provide a stereographic effect.

17. The method of claim 1, further comprising the step of employing a graphic rendering algorithm to blur objects that are not disposed at the apparent focus plane in the image.

18. The method of claim 1, wherein the step of determining the accommodation comprises the step of employing light that is not visible to a human, to measure the accommodation for the eye of the viewer.

19. The method of claim 1, wherein the image that is displayed by the large depth of focus display is in a non-planar format.

42. The method of claim 12, further comprising the step of rendering a 2-dimensional image at a desired apparent focus plane using the depth information for the 2½

-dimensional image, if the accommodation of the viewer has changed, thereby reducing a computational overhead because the 2½

-dimensional image does not have to be re-rendered.

43. The method of claim 12, further comprising the step of re-rendering the 2½

-dimensional image if either the accommodation of the viewer moves within a scene or the viewer views an object that is moving in the scene.

44. The method of claim 12, wherein the 2½

-dimensional image is either a cylindrical image or a spherical image.

20. A system for more accurately conveying depth in an image, comprising:

(a) a large depth of focus display;

(b) an image source that cooperates with the large depth of focus display to produce an image that can be viewed;

(c) a device that monitors at least one eye of a viewer to produce a signal indicative of an accommodation of said at least one eye; and

(d) a computing device coupled to the image source and to the device, said computing device carrying out a plurality of functions, including;

(i) displaying the image to a viewer on the large depth of focus display wherein all elements in the image are initially displayed at an optical focus level that is substantially the same for all elements, thereby simulating an unnatural or artificial viewing condition, since elements viewed at different distances from the viewer would naturally appear at different focus levels;

(ii) determining an accommodation for an eye of a viewer who is watching the image on the large depth of focus display as a gaze of the viewer is directed toward an element in the image; and

(iii) displaying an image having an apparent focus plane that tracks the accommodation of the viewer, so that as the accommodation of the viewer by adding blurring to other elements in the image in proportion to their distance in depth from the viewpoint of the viewer, so that as the accommodation of the viewer watching the large depth of focus display changes, to focus the eye of the viewer at a different viewing distance, the image that is displayed is changed to more accurately visually convey depth in the image that is displayed, based on the accommodation that was determined, thereby simulating a natural viewing condition.

21. The system of claim 20, wherein the device emits light for directly measuring the accommodation in at least one eye of the viewer.

22. The system of claim 20, wherein the device determines the accommodation by:

(a) measuring a vergence of at least one eye of the viewer; and

(b) determining the accommodation as a function of the vergence.

23. The system of claim 20, wherein the device measures a gaze direction of the viewer, and the computing device anticipates the accommodation of the viewer based upon the gaze direction.

24. The system of claim 20, wherein in real-time, the computing device renders each image having an apparent focus plane that tracks the accommodation of the viewer, on the large depth of focus display.

25. The system of claim 24, wherein objects within each image that are farther away from the apparent focus plane in the image are rendered at a lower resolution and contrast by the computing device, to substantially reduce a computational overhead required for rendering the image on the large depth of focus display.

26. The system of claim 20, wherein a plurality of images having a range of different apparent focus planes are pre-prepared, so that the image having the apparent focus plane that tracks the accommodation of the viewer is selected by the computing device from the plurality of images that were pre-prepared.

27. The system of claim 26, wherein the plurality of images are arranged in a multi-dimensional array, at least one axis of the multi-dimensional array corresponding to a disposition of the apparent focus plane in the plurality of images.

28. The system of claim 27, wherein each other dimension of the multi-dimensional array corresponds to a different parameter that varies within the plurality of images.

29. The system of claim 28, wherein the computing device responds to a user input that varies a value of a parameter for at least one of the other dimensions, causing a corresponding change in the image on the large depth of focus display.

30. The system of claim 29, wherein the image source comprises a camera that is used to produce the plurality of images, and wherein the parameter comprises one of:

(a) a motion of the camera into a scene comprising the plurality of images;

(b) an orientation of the camera when imaging a scene to produce the plurality of images; and

(c) a zoom level of the camera when producing the plurality of images.

31. The system of claim 27, wherein the image source displays a 2½

-dimensional image on the large depth of focus display, so that a 2-dimensional image can be rendered by the computing device at a desired apparent focus plane using the depth information for the 2½

-dimensional image, to reduce computational overhead.

32. The system of claim 25, wherein the image source comprises a camera having a variable focus, and wherein the plurality of images are pre-prepared by capturing a scene with the camera with the variable focus set at a plurality of different focal planes.

33. The system of claim 20, further comprising an actuator coupled to a variable focus adjustment of a camera and to the computing device, said computing device producing the image having the apparent focus plane that tracks the accommodation of the viewer by controlling the actuator to adjust a focus of the camera so that the camera produces said image by imaging a real scene with the focus set at said apparent focus plane.

34. The system of claim 20, wherein the computing device selects successive images having apparent focus planes that track the accommodation of the viewer, at a sufficiently fast image rate to produce a perception of motion of an object within the successive images viewed on the large depth of focus display.

35. The system of claim 20, further comprising another image source that is coupled to the computing device and produces an image in which at least one element is laterally shifted, said image having an apparent focus plane that tracks the accommodation of the viewer, so that each eye sees a different image, to provide a stereographic effect.

36. The system of claim 20, wherein the computing device executes a graphic rendering algorithm to blur objects that are not disposed at the apparent focus plane in the image.

37. The system of claim 20, wherein the device uses light that is not visible to a human to measure the accommodation for the eye of the viewer.

38. The system of claim 20, wherein the image source displays an image on the large depth of focus display in a non-planar format.

39. The system of claim 20, further comprising a beam splitter so that light from the image source is reflected into an eye of the viewer, while light used by the device for determining the accommodation travels between the device and the eye of the viewer through the beam splitter.

40. The system of claim 20, further comprising a beam splitter so that light from the image source is transmitted into an eye of the viewer, while light used by the device for determining the accommodation is reflected into the eye of the viewer by the beam splitter.

41. The system of claim 20, further comprising a beam splitter, said beam splitter reflecting light from one of the image source and a real world scene, so that the viewer can simultaneously view the real world scene and the image provided by the image source.

45. A method for more accurately conveying depth in an image, comprising the steps of:

(a) displaying a 2½

-dimensional image comprising visual depth information to a viewer on a large depth of focus display;

(b) determining an accommodation for an eye of the viewer who is watching the 2½

-dimensional image on the large depth of focus display;

(c) preparing a plurality of images arranged in a multidimensional array and having a range of different apparent focus planes, at least one axis of the multi-dimensional array corresponding to a disposition of the apparent focus plane in the plurality of images;

(d) selecting the image having the apparent focus plane that tracks the accommodation of the viewer from the plurality of images that were pre-prepared, said image being a 2-dimensional image; and

(e) as the accommodation of the viewer watching the large depth of focus display changes, using the depth information of the 2½

-dimensional image to display the 2-dimensional image, in order to more accurately visually convey depth, said use of depth information thereby reducing a computational overhead because the 2½

-dimensional image does not have to be re-rendered.

46. A method for more accurately conveying depth in an image, comprising the steps of:

(a) displaying a 2½

-dimensional image comprising visual depth information to a viewer on a large depth of focus display;

(b) determining an accommodation for an eye of the viewer who is watching the 2½

-dimensional image on the large depth of focus display;

(c) preparing a plurality of images arranged in a multidimensional array and having a range of different apparent focus planes, at least one axis of the multi-dimensional array corresponding to a disposition of the apparent focus plane in the plurality of images;

(d) selecting the image having the apparent focus plane that tracks the accommodation of the viewer from the plurality of images that were pre-prepared, said image being a 2½

-dimensional image; and

(e) as the accommodation of the viewer watching the large depth of focus display moves within a scene or the viewer views an object that is moving in the scene, displaying the 2½

-dimensional image having the apparent focus plane that tracks the accommodation of the viewer, in order to more accurately visually convey depth, based on the accommodation that was determined.