Apparatus and method for a dynamic “region of interest” in a display system

• US 9,618,748 B2
• Filed: 09/27/2010
• Issued: 04/11/2017
• Est. Priority Date: 04/02/2008
• Status: Active Grant

• Current Assignee: eSight Corporation
• Original Assignee: eSight Corporation
• Inventors: Munger, Rejean J. Y. B., Hilkes, Robert G., Perron, Marc, Sohi, Nirmal
• Primary Examiner(s): CONNER, SEAN M
• Application Number:
  US12/891,430
• Publication Number:
  US 20110043644A1
• Time in Patent Office: 2,388 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/346 using prisms or semi-transparent mirrors
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A method and apparatus of displaying a magnified image comprising obtaining an image of a scene using a camera with greater resolution than the display, and capturing the image in the native resolution of the display by either grouping pixels together, or by capturing a smaller region of interest whose pixel resolution matches that of the display. The invention also relates to a method whereby the location of the captured region of interest may be determined by external inputs such as the location of a person'"'"'s gaze in the displayed unmagnified image, or coordinates from a computer mouse. The invention further relates to a method whereby a modified image can be superimposed on an unmodified image, in order to maintain the peripheral information or context from which the modified region of interest has been captured.

Claims

1. A method of augmenting sight in an individual, the method comprising the steps of:

obtaining a field of view image of a scene to be viewed by the individual using a camera carried by the individual;

transmitting the obtained field of view image to a processor carried by the individual;

receiving from the individual an indication of a selected region of interest within the field of view image, the selected region of interest defined by the individual and not aligned to the visual axis of the individual;

receiving from the individual an indication of a selected image modification to be applied to the region of interest within the field of view image by the processor, the selected image modification being selected by the individual from a plurality of selectable image modifications;

receiving data relating to a visual dysfunction relating to a visual field of view of the individual, a predetermined portion of the data defining damaged areas of the individual'"'"'s visual field of view to avoid;

operating, by the processor, upon the region of interest within the field of view image to create a modified region of interest within the field of view image in response to the selected image modification; and

displaying the modified region of interest within the field of view image on a display device forming part of a head worn assembly, wherein the displaying comprises relocating pixels of the damaged areas to areas near a periphery of the damaged areas according to a spatial distribution model.

2. The method according to claim 1, wherein receiving from the individual the indication of the selected region of interest comprises:

receiving a first indication relating to at least one of the size, shape, and location of the region of interest within the field of view image;

receiving a second indication relating to at least one of the size, shape, and location of the modified region of interest to be displayed to the individual.

3. The method according to claim 1, wherein

the head worn assembly worn by the individual comprises an optical prism allowing the individual to view both the field of view image and the modified region of interest.

4. The method according to claim 1, further comprising

receiving from the individual an indication of a second selected image modification to be applied to the field of view image by the processor, the second selected image modification being selected by the individual from a plurality of selectable image modifications;

operating, by the processor, upon the field of view image to create a modified field of view image in response to the second selected image modification; and

displaying the modified field of view image as the field of view image within which the modified region of interest is displayed.

5. The method according to claim 1, wherein

the head worn assembly worn by the individual comprises display devices for left and right eyes of the individual;

and

displaying the modified region of interest within the field of view image is performed for both the left and right eyes;

wherein

receiving from the individual an indication of a selected image modification comprises receiving either a selected image modification that is applied equally to both the left and right eyes or a selected image modification that is applied to the left and right eyes differently.

6. The method according to claim 1, wherein receiving from the individual the indication of the selected image modification comprises receiving the selected image modification that is applied to the region of interest and a second selected image modification that is applied to the field of view image.

7. The method according to claim 1, further comprising receiving from the individual an indication of a degree of image modification, the degree of image modification relating to the selected image modification;

and operating upon the region of interest to create the modified region of interest in response to the selected image modification and the indication of the degree of the image modification.

8. A method of augmenting sight in an individual, the method comprising the steps of:

obtaining a field of view image of a scene to be viewed by the individual using a camera carried by the individual;

transmitting the obtained field of view image to a processor carried by the individual;

receiving from the individual an indication of a selected region of interest within the field of view image;

receiving data relating to a visual dysfunction relating to the visual field of view of the individual, a predetermined portion of the data defining damaged areas of the individual'"'"'s visual field of view to avoid;

receiving an indication of a gaze direction of the individual;

establishing a selected image modification to be applied to the region of interest within the field of view image by the processor, the selected image modification being selected from a plurality of selectable image modifications by the individual;

modifying substantially in real time, with the processor, the region of interest to create a modified region of interest in response to the selected image modification, the selected region of interest, the gaze direction of the individual and the predetermined portion of the data relating to the visual dysfunction that defines the damaged areas of the individual'"'"'s visual field of view to avoid; and

displaying the field of view image and the modified region of interest on a display device forming part of a head worn assembly worn by the individual, wherein the displaying comprises relocating pixels of the damaged areas to areas near a periphery of the damaged areas according to a spatial distribution model.

9. The method according to claim 8, wherein receiving from the individual the indication of the selected region of interest within the field of view image comprises:

receiving a first indication relating to whether the selected region of interest is within the field of view image;

receiving a second indication relating to at least one of the size, shape, and location of the region of interest within the field of view image; and

receiving a third indication relating to at least one of the size and shape of the modified region of interest to be displayed to the individual together with a preferred location of the modified region of interest to be displayed.

10. The method according to claim 8, wherein receiving from the individual the indication of the selected region of interest within the field of view image comprises:

receiving a first indication relating to whether the selected region of interest is within the field of view image;

receiving a second indication relating to at least one of the size, shape, and location of the region of interest within the field of view image; and

receiving a third indication relating to at least one of the size, shape, and location of the modified region of interest to be displayed to the individual.

11. The method according to claim 8, wherein

the head worn assembly worn by the individual comprises an optical pipeline allowing the individual to view both the field of view image and the modified region of interest and a portion of their field of view perceived directly at least one of around and through the optical pipeline.

12. A method of augmenting sight in an individual, the method comprising the steps of:

receiving from the individual an indication of a selected region of interest within their field of view;

obtaining a region of interest image relating to the selected region of interest from an image acquired using a camera carried by the individual;

transmitting the obtained region of interest image to a processor carried by the individual;

receiving data relating to a visual dysfunction relating to the visual field of view of the individual, a predetermined portion of the data defining damaged areas of the individual'"'"'s visual field of view to avoid;

receiving an indication of a gaze direction of the individual;

establishing a selected image modification to be applied to the region of interest by the processor, the selected image modification being selected from a plurality of selectable image modifications;

modifying substantially in real time, with the processor, the region of interest image to create a modified region of interest image in response to the selected image modification, the gaze direction of the individual and the predetermined portion of the data relating to the visual dysfunction that defines the damaged areas of the individual'"'"'s visual field of view to avoid; and

displaying the modified region of interest image on a display device forming part of a head worn assembly worn by the individual, wherein the displaying comprises relocating pixels of the damaged areas to areas near a periphery of the damaged areas according to a spatial distribution model.

13. The device according to claim 12, wherein

displaying the modified region of interest on the display device comprises displaying the modified region of interest together with image data relating to their field of view captured by a second camera carried by the individual.

14. The device according to 13, wherein the second camera and the camera carried by the individual are the same camera.

15. The device according to claim 12, wherein

the head worn assembly worn by the individual comprises an optical prism allowing the individual to view both a portion of their field of view through the optical prism and the modified region of interest displayed upon the display.

16. The method according to claim 12, wherein receiving from the individual the indication of the selected region of interest within the region of interest image comprises:

receiving a first indication relating to a preferred location of displaying the selected region of interest within the region of interest image.

17. A method of augmenting sight in an individual, the method comprising the steps of:

obtaining a field of view image of a scene to be viewed by the individual using a camera carried by the individual;

transmitting the obtained field of view image to a processor carried by the individual;

receiving data relating to a visual dysfunction relating to the visual field of view of the individual, a predetermined portion of the data defining damaged areas of the individual'"'"'s visual field of view to avoid;

receiving an indication of at least one of a gaze direction of the individual;

receiving from the individual either an indication of a method of algorithmically selecting a region of interest or an indication of a region of interest, wherein the region of interest relates to the field of view image and the method is one of a plurality of such methods;

establishing a selected image modification to be applied to the region of interest by the processor, the selected image modification being selected from a plurality of selectable image modifications;

modifying substantially in real time, with the processor, the region of interest to create a modified region of interest image in response to the selected image modification, the selected region of interest, the gaze direction of the individual, and the predetermined portion of the data relating to the visual dysfunction that defines the damaged areas of the individual'"'"'s visual field of view to avoid; and

displaying the field of view image and the modified region of interest image on a display device forming part of a head worn assembly worn by the individual, wherein the displaying comprises relocating pixels of the damaged areas to areas near a periphery of the damaged areas according to a spatial distribution model.

18. The method according to claim 17, wherein the plurality of selectable image modifications varies in dependence upon the method of algorithmically selecting the region of interest.