Method and apparatus for a dynamic “region of interest” in a display system

• US 9,720,238 B2
• Filed: 05/25/2016
• Issued: 08/01/2017
• Est. Priority Date: 04/02/2007
• Status: Active Grant

• Current Assignee: Gentex Corporation
• Original Assignee: eSight Corporation (Gentex Corporation)
• Inventors: Hilkes, Robert, Perron, Marc, Sohi, Nirmal, Munger, Rejean
• Primary Examiner(s): CONNER, SEAN M
• Application Number:
  US15/163,790
• Publication Number:
  US 20160282624A1
• Time in Patent Office: 433 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/346 using prisms or semi-transparent mirrors
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A method and apparatus of displaying a magnified image comprising obtaining an image of a scene using a camera with greater resolution than the display, and capturing the image in the native resolution of the display by either grouping pixels together, or by capturing a smaller region of interest whose pixel resolution matches that of the display. The invention also relates to a method whereby the location of the captured region of interest may be determined by external inputs such as the location of a person'"'"'s gaze in the displayed unmagnified image, or coordinates from a computer mouse. The invention further relates to a method whereby a modified image can be superimposed on an unmodified image, in order to maintain the peripheral information or context from which the modified region of interest has been captured.

Claims

1. A method of augmenting sight for a user, the method comprising the steps of:

obtaining a field of view image of a scene viewed by the user using a forward-facing two-dimensional image-capture device carried by the user;

transmitting the obtained image to a processor carried by the user;

selecting a region of interest within either the field of view image or a second image;

selecting an image modification to be applied to the region of interest by the processor;

operating, by the processor, upon the region of interest to create a modified region of interest image in response to the selected image modification;

digitally combining the field of view image and the modified region of interest image such that the modified region of interest image replaces image content from the location of the region of interest of the field of view image to form a hybrid image; and

displaying the hybrid image on a two-dimensional display device carried by the user;

wherein the field of view image is one of a plurality of field of view images captured by the forward-facing two-dimensional image-capture device and transmitted to the processor at a first frame rate;

the region of interest image is one of a plurality of region of interest images transmitted to the processor at a second frame rate; and

the processor generates a plurality of hybrid display images by combining the plurality of field of view images at the first frame rate and the modified plurality of region of interest images at the second frame rate.

2. The method according to claim 1, wherein

the user cannot view the scene directly.

3. The method according to claim 1, wherein

the camera, processor, and display device form part of a head worn device detachably mounted to a frame carrying either a prescription lens or a pair of prescription lenses worn in the same manner as a pair of spectacles such that the head worn display is viewed by the user through the one of the prescription lens and the pair of prescription lenses when mounted to the frame.

4. The method according to claim 1, wherein

the processor is an electronic device wirelessly connected to both the display device and the camera, and

the display device forms part of a head worn device display together with a frame and either a prescription lens or a pair of prescription lenses such that the head worn display is worn in the same manner as a pair of spectacles and the display device is viewed by the user through the one of the prescription lens and the pair of prescription lenses.

5. The method according to claim 1, wherein

selecting the region of interest comprises:

receiving a first indication relating to whether the selected region of interest is within the field of view image or from the separate region of interest image source;

receiving a second indication relating to at least one of the size, shape, and location of the region of interest within the field of view image or from the separate region of interest image source;

receiving a third indication relating to at least one of the size and shape of the modified region of interest to be displayed to the individual together with a preferred location of the modified region of interest to be displayed.

6. The method according to claim 1, wherein

the region of interest is determined by an eye tracking system employing averaging and damping established in dependence upon a usage model selected based upon the user'"'"'s current activity.

7. A head worn system augmenting sight for a user comprising:

a forward-facing two-dimensional image-capture device, carried by the user, for obtaining a field of view image of a scene associated with the user;

a processor, carried by the user, the processor in communication with the two-dimensional image-capture device and a display, the processor for;

establishing an indication of a region of interest within either the field of view image or a second image;

establishing an image modification to be applied to the region of interest by the processor;

modifying the region of interest to create a modified region of interest image;

digitally combining the field of view image and the modified region of interest image such that the modified region of interest image replaces image content from the location of the region of interest of the field of view image to form a hybrid image; and

transmitting the hybrid image to a two-dimensional display; and

the two-dimensional display which forms part of the head worn system to be worn by the user,

wherein the field of view image is one of a plurality of field of view images captured by the forward-facing two-dimensional image-capture device and transmitted to the processor at a first frame rate;

the region of interest image is one of a plurality of region of interest images transmitted to the processor at a second frame rate; and

the processor generates a plurality of hybrid display images by combining the plurality of field of view images at the first frame rate and the modified plurality of region of interest images at the second frame rate.

8. The head worn system according to claim 7, wherein

the user cannot view the scene directly.

9. The head worn system according to claim 7, wherein

the two-dimensional image-capture device and processor are wirelessly connected;

and

the head worn system further comprises;

a frame such that the head worn system is worn in the same manner as a pair of spectacles; and

either a prescription lens or a pair of prescription lenses such that the two-dimensional image-capture device is viewed by the user through the one of the prescription lens and the pair of prescription lenses.

10. The head worn system according to claim 7, wherein

the two-dimensional image-capture device and processor also form part of the head worn system which is detachably mounted to a frame carrying either a prescription lens or a pair of prescription lenses worn in the same manner as a pair of spectacles such that the head worn display is viewed by the user through the one of the prescription lens and the pair of prescription lenses when mounted to the frame.

11. The head worn system according to claim 7, wherein

selecting the region of interest comprises:

receiving a first indication relating to whether the selected region of interest is within the field of view image or from the separate region of interest image source;

receiving a second indication relating to at least one of the size, shape, and location of the region of interest within the field of view image or from the separate region of interest image source;

receiving a third indication relating to at least one of the size and shape of the modified region of interest to be displayed to the individual together with a preferred location of the modified region of interest to be displayed.

12. The head worn system according to claim 7, wherein

the region of interest is determined by an eye tracking system employing averaging and damping established in dependence upon a usage model selected based upon the user'"'"'s current activity.