High resolution perception of content in a wide field of view of a head-mounted display

• US 10,007,333 B2
• Filed: 11/09/2015
• Issued: 06/26/2018
• Est. Priority Date: 11/07/2014
• Status: Active Grant

• Current Assignee: Eye Labs, LLC
• Original Assignee: Eye Labs, LLC
• Inventors: Fateh, Sina
• Primary Examiner(s): Thompson, James A
• Application Number:
  US14/936,540
• Publication Number:
  US 20160133055A1
• Time in Patent Office: 960 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0202 Constructional details or processes of manufacture of the input device
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/147 : using display panels
• CPC Class Codes: G06T 13/20 : 3D [Three Dimensional] animation
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2200/24 : involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20172 : Image enhancement details
• CPC Class Codes: G06T 2210/36 : Level of detail
• CPC Class Codes: G06T 3/40 : Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G09G 2340/0407 : Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/0485 : Centering horizontally or vertically
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 2213/002 : Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices

Abstract

Introduced herein are various techniques for displaying virtual and augmented reality content via a head-mounted display (HMD). The techniques can be used to improve the effectiveness of the HMD, as well as the general experience and comfort of users of the HMD. An HMD may increase and/or decrease the resolution of certain areas in digital content that is being viewed to more accurately mimic a user'"'"'s high resolution and low resolution fields of view. For example, the HMD may monitor the user'"'"'s eye movement to identify a focal point of the user'"'"'s gaze, and then increase the resolution in an area surrounding the focal point, decrease the resolution elsewhere, or both. Predictive algorithms could also be employed to identify which areas are likely to be the subject of the user'"'"'s gaze in the future, which allows the HMD to present the regionally-focused content in real-time.

Claims

1. A system for regionally modifying resolution of digital content to emulate real-world fields of view, the system comprising:

a processor communicatively coupled to a head-mounted device operated by a user and operable to execute instructions stored in a memory; and

the memory, which includes specific instructions for regionally modifying the resolution of digital content displayed by the head-mounted device based on a location of a focal point of the user'"'"'s gaze,

wherein execution of the specific instructions causes the processor to;

cause digital content to be displayed by the head-mounted device;

identify a location of the focal point at a first point in time;

identify a first area of the digital content that includes the focal point at the first point in time;

increase resolution of the first area of the digital content;

identify a second area of the digital content that is likely to include the focal point at a second point in time;

increase resolution of the second area of the digital content; and

decrease resolution of a third area of the digital content that is located between the first and second areas of the digital content,

wherein the resolutions of the first, second, and third areas are modified substantially simultaneously.

2. The system of claim 1, wherein execution of the specific instructions further causes the processor to:

continuously monitor the focal point in real time;

determine whether the focal point has changed; and

responsive to a determination that the focal point has changed, modifying a size or a position of the first area of the digital content.

3. The system of claim 2, wherein said continuously monitoring includes monitoring movement of the user'"'"'s eyes, head, or both.

4. The system of claim 1, wherein the resolution of the third area is decreased proportional to eye movement speed, head movement speed, or some combination thereof.

5. The system of claim 1, wherein the third area is the remainder of the digital content that is not included in the first area or the second area.

6. The system of claim 1, wherein the processor uses predictive algorithms to identify the second area of the digital content that is likely to include the focal point at the second point in time.

7. The system of claim 6, wherein the predictive algorithms consider speed and direction of the user'"'"'s head movement, a digital element of the digital content, or both.

8. A head-mounted device comprising:

a supportive frame;

an optical display surface coupled to the supportive frame;

an image display system configured to cause display of digital content on the optical display surface;

a first sensor configured to continuously track a focal point of a user'"'"'s gaze in real time;

a second sensor configured to continuously monitor eye movement speed or head movement speed;

a processor, communicatively coupled to the image display system, configured to execute instructions stored in a memory; and

the memory, which includes specific instructions for regionally modifying resolution of the digital content displayed on the optical display surface, wherein execution of the specific instructions causes the processor to;

cause a first digital image to be displayed on the optical display surface;

identify a location of the focal point at a first point in time;

identify a first area of the first digital image that includes the focal point at the first point in time;

increase resolution of the first area of the first digital image;

identify a second area of the first digital image that does not include the focal point at the first point in time; and

decrease resolution of the second area of the first digital image,

wherein the decrease in resolution of the second area of the first digital image is proportional to the eye movement speed or the head movement speed, and

wherein the resolutions of the first and second area are modified substantially simultaneously.

9. The head-mounted device of claim 8, wherein the changes in resolution of the first and second areas are performed for the purpose of mimicking a high-resolution viewing field and a low-resolution viewing field naturally experienced by the user when viewing a real-world environment.

10. The head-mounted device of claim 8, wherein the second area includes a series of sections, and wherein the resolution of each section is decreased proportional to the distance between the section and the focal point of the user'"'"'s gaze.

11. The head-mounted device of claim 8, further comprising:

a third sensor configured to detect gestures performed by the user,

wherein said identifying the location of the focal point at the first point in time is based at least in part on a gesture performed by the user.

12. The head-mounted device of claim 8, wherein execution of the specific instructions further causes the processor to:

identify a candidate area of a second digital image that is likely to become the subject of the user'"'"'s gaze at a second point in time using a predictive algorithm,

wherein the candidate area is part of the second area of the first digital image at the first point in time.

13. The head-mounted device of claim 12, wherein machine learning techniques are employed to periodically improve the predictive algorithm.

14. The head-mounted device of claim 12, wherein the predictive algorithm considers as input direction of the user'"'"'s head movement, speed of the user'"'"'s head movement, or both.

15. The head-mounted device of claim 12, wherein the predictive algorithm considers as input a digital element of the second digital image to be displayed at the second point in time and that is determined to likely become the subject of the user'"'"'s gaze.

16. A method for regionally modifying resolution of digital content that is presented to a user of a head-mounted device, the method comprising:

causing a head-mounted device to display digital content for viewing by a user;

determining where a focal point of the user'"'"'s gaze is located at a first point in time;

identifying a first area of an optical display surface that includes the focal point at the first point in time;

increasing resolution of a first digital image within the first area;

identifying a second area of the optical display surface that is determined to likely include the focal point at a second point in time at which a second digital image is shown;

determining that the focal point has begun to move toward the second area of the optical display surface;

increasing resolution of the second digital image within the second area;

identifying a third area of the optical display surface located between the first area and the second area; and

decreasing resolution of the first and second digital images within the third area;

wherein the resolutions of the first and second digital images are modified substantially simultaneously as the focal point moves toward the second area of the optical display surface.

17. The method of claim 16, wherein the third area is the remainder of the optical display surface not included in the first area and the second area.

18. The method of claim 16, wherein the third area is composed of a series of sections that each include a plurality of pixels, and wherein the resolution of each section is decreased proportional to the distance between the section and the first area or the second area.

19. The method of claim 16, wherein the second area of the optical display surface is located in a fixed, predetermined location.

20. The method of claim 19, wherein the fixed, predetermined location corresponds to a feature of the digital content that is always or nearly always viewable by the user.