Apparatus and Method for a Dynamic "Region of Interest" in a Display System

• US 20110043644A1
• Filed: 09/27/2010
• Published: 02/24/2011
• Est. Priority Date: 04/02/2008
• Status: Active Grant

• Current Assignee: eSight Corporation
• Original Assignee: eSight Corporation
• Inventors: Hilkes, Robert G., Perron, Marc, Sohi, Nirmal, Munger, Rejean J. Y. B.
• Granted Patent: US 9,618,748 B2
• Time in Patent Office: Days
• US Class Current: 348/207.1
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/346 using prisms or semi-transparent mirrors
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A method and apparatus of displaying a magnified image comprising obtaining an image of a scene using a camera with greater resolution than the display, and capturing the image in the native resolution of the display by either grouping pixels together, or by capturing a smaller region of interest whose pixel resolution matches that of the display. The invention also relates to a method whereby the location of the captured region of interest may be determined by external inputs such as the location of a person'"'"'s gaze in the displayed unmagnified image, or coordinates from a computer mouse. The invention further relates to a method whereby a modified image can be superimposed on an unmodified image, in order to maintain the peripheral information or context from which the modified region of interest has been captured.

Claims

1. A method of obtaining an image whereby the number of pixels in the captured image is selected to exactly match the resolution of the display comprising the steps of:

obtaining an image of a scene viewed by the individual using a camera that has higher resolution than the display;

capturing an image from the camera by combining pixels such that the resolution of the captured image exactly matches that of the display; and

displaying the image on a display device.

2. The method of claim 1 wherein the pixels are combined by adjusting the camera resolution.

3. A method of displaying a magnified image with neither the use of optical zoom lenses nor software magnification algorithms comprising the steps of:

obtaining an image of a scene viewed by the individual using a camera that has higher resolution than the display;

capturing a region of interest from the camera that is less than the full field of view of the camera, and that exactly matches the pixel resolution of the display;

displaying the image on a display device.

4. The method of claim 3 wherein the displayed region of interest is overlayed on top of a full field of view image displayed.

5. The method of claim 4 wherein the frame capture rate of the full field of view image is lower than that of the region of interest, such that the combined images when overlayed, provide a high quality magnified image of the region of interest, with a lower quality unmagnified image of the full field of view.

6. The method of claim 3 wherein the location of the region of interest in the camera is determined by a person'"'"'s gaze location in the displayed image.

7. The method of claim 3 wherein the location of the region of interest in the camera is determined by another input device.

8. The method of claim 3 wherein the region of interest and the field of view are captured from the same image sensor sequentially.

9. A method of claim 3 wherein the region of interest and the field of view are captured from two separate cameras, for viewing on a single display.

10. A method of augmenting sight in an individual, the method comprising the steps of:

obtaining a field of view image of a scene viewed by the individual using a camera carried by the individual;

transmitting the obtained image to a processor carried by the individual;

selecting a region of interest within the field of view image;

selecting an image modification to be applied to the region of interest within the field of view image by the processor; and

operating, by the processor, upon the region of interest within the field of view image to create a modified region of interest within the field of view image in response to the selected image modification; and

displaying the modified region of interest within the field of view image on a display device carried by the individual.

11. The method of claim 10 wherein the image modification is magnification.

12. The method of claim 10 wherein the image modification is a remapping of the colors within the region of interest in the field of view image.

13. The method of claim 10 wherein the image modification is edge enhancement.

14. The method of claim 10 wherein the image modification is at least one of brightness and contrast intensity enhancement.

15. The method of claim 10 wherein the image modification occurs substantially in real time.

16. The method of claim 10 wherein the image modification is alternatively applied and removed.

17. An apparatus for augmenting sight in an individual comprising:

a camera, carried by the individual, for obtaining an image of a scene viewed by the individual;

a display carried by the individual;

an image modification input device carried by the individual; and

a processor, carried by the individual, the processor in communication with the camera, image modification input device and display, the processor for modifying a region of interest in the image obtained by the camera, in response to the instructions provided by the individual using the image modification input device, and for displaying a modified region of interest image on the display carried by the individual.

18. A method for improving the visual acuity of a patient with retinal defects comprising the steps of:

determining the locations of retinal damage in an eye of the patient;

obtaining a region of interest in an image of a scene viewed by the patient; and

mapping the region of interest image to a display in such a way to avoid the locations of retinal damage when the display is viewed by the patient.

19. The method of claim 18 further comprising modifying the region of interest image using an operation selected from a set of operations comprising magnify, edge enhance, remap color, remap visual field, contrast enhance and brightness enhance the image.

20. A method of augmenting sight in an individual, the method comprising the steps of:

obtaining a field of view image of a scene viewed by the individual using a camera carried by the individual;

transmitting the obtained image to a processor carried by the individual;

selecting a region of interest within the field of view image;

selecting an image modification to be applied to the region of interest within the field of view image by the processor; and

displaying the modified region of interest on a display device that is otherwise transparent, carried by the individual; and

while displaying the region of interest at a specific location of the display, enabling the individual to look through the transparent portion of the display outside the boundary of the region of interest, in order to maintain their peripheral vision.

21. A method of improving the performance of a head worn display by incorporating the wearer'"'"'s prescription lenses into the frame of the device.