High resolution perception of content in a wide field of view of a head-mounted display

• US 9,804,669 B2
• Filed: 11/09/2015
• Issued: 10/31/2017
• Est. Priority Date: 11/07/2014
• Status: Expired due to Fees

• Current Assignee: Eye Labs, LLC
• Original Assignee: Eye Labs, LLC
• Inventors: Fateh, Sina
• Primary Examiner(s): BEUTEL, WILLIAM A
• Application Number:
  US14/936,533
• Publication Number:
  US 20160133170A1
• Time in Patent Office: 722 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0202 Constructional details or processes of manufacture of the input device
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/147 : using display panels
• CPC Class Codes: G06T 13/20 : 3D [Three Dimensional] animation
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2200/24 : involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20172 : Image enhancement details
• CPC Class Codes: G06T 2210/36 : Level of detail
• CPC Class Codes: G06T 3/40 : Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G09G 2340/0407 : Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/0485 : Centering horizontally or vertically
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 2213/002 : Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices

Abstract

Introduced herein are various techniques for displaying virtual and augmented reality content via a head-mounted display (HMD). The techniques can be used to improve the effectiveness of the HMD, as well as the general experience and comfort of users of the HMD. An HMD may increase and/or decrease the resolution of certain areas in digital content that is being viewed to more accurately mimic a user'"'"'s high resolution and low resolution fields of view. For example, the HMD may monitor the user'"'"'s eye movement to identify a focal point of the user'"'"'s gaze, and then increase the resolution in an area surrounding the focal point, decrease the resolution elsewhere, or both. Predictive algorithms could also be employed to identify which areas are likely to be the subject of the user'"'"'s gaze in the future, which allows the HMD to present the regionally-focused content in real-time.

Claims

1. A system for regionally modifying a resolution of digital content to create a more natural viewing experience that emulates real-world fields of view, the system comprising:

a processor communicatively coupled to a head-mounted device operated by a user and operable to execute instructions stored in a memory; and

the memory, which includes specific instructions for modifying a resolution of digital content presented by the head-mounted device based on a location of a focal point of the user'"'"'s gaze,

wherein execution of the specific instructions prompts the processor to;

cause a digital image to be displayed by the head-mounted device;

continuously monitor the focal point of the user'"'"'s gaze in real time, the focal point being somewhere within the bounds of the digital image;

determine where the focal point is located at a first point in time;

identify a first area of the digital image that includes the focal point of the user'"'"'s gaze at the first point in time;

increase resolution of the first area of the digital image;

determine that the focal point is going to change based on a digital element of another digital image to be subsequently shown at a second point in time;

monitor a speed and a direction of the user'"'"'s eye movement and head movement;

identify a second area of the digital image that is predicted to include the focal point of the user'"'"'s gaze at the second point in time;

increase resolution of the second area of the digital image; and

decrease resolution of a third area of the digital image located between the first and second areas of the digital image,

where the resolutions of the first area, the second area, and the third area are modified substantially simultaneously as the digital content transitions from the digital image to the other digital image,

wherein a size of the first area and a size of the second area are based on the speed of the user'"'"'s eye movement, the speed of the user'"'"'s head movement, or a combination thereof, and

where the decrease in resolution of the third area of the digital image is proportional to the speed of the user'"'"'s eye movement, the speed of the user'"'"'s head movement, or a combination thereof.

2. The system of claim 1, wherein the first area is circular in shape and the focal point is located at the center of the first area.

3. The system of claim 1, wherein the increases in resolution of the first area and the second area are proportional to the speed of the user'"'"'s eye movement, the speed of the user'"'"'s head movement, or both.

4. The system of claim 3, wherein the specific instructions are further configured to:

employ a predictive algorithm to identify at least one area of the digital image likely to include the focal point of the user'"'"'s gaze at a future time.

5. The system of claim 4, wherein the predictive algorithm considers the speed and the direction of the user'"'"'s head movement and eye movement.

6. The system of claim 1, wherein the third area corresponds to the user'"'"'s natural low resolution viewing field.

7. A head-mounted device configured to present regionally-focused digital content that mimics a user'"'"'s natural high-resolution and low-resolution fields of view, the head-mounted device comprising:

a supportive frame;

an optical display surface coupled to the supportive frame;

an image display system configured to project digital content onto the optical display surface;

a first sensor coupled to the supportive frame and configured to continuously track a focal point of the user'"'"'s gaze in real time;

a second sensor coupled to the supportive frame and configured to continuously monitor speed, direction, or both of the user'"'"'s head movement;

a third sensor coupled to the supportive frame and configured to detect gestures performed by the user;

a processor communicatively coupled to the image display system and operable to execute instructions stored in a memory; and

the memory, which includes specific instructions for regionally modifying a resolution of digital images shown on the optical display surface, wherein execution of the specific instructions prompts the processor to;

cause a first digital image to be displayed on the optical display surface;

continuously track the focal point of the user'"'"'s gaze within the bounds of the first digital image;

determine where the focal point is located at a first point in time;

identify a first area of the first digital image that includes the focal point of the user'"'"'s gaze at the first point in time,

where a size of the first area is based on the speed of the user'"'"'s eye movement, the speed of the user'"'"'s head movement, or a combination thereof;

increase resolution of the first area of the first digital image;

identify a second area of the first digital image that does not include the focal point of the user'"'"'s gaze but is predicted to include the focal point of the user'"'"'s gaze at a second point in time,

where the second area is entirely distinct from the first area,

where a size of the second area is based on the speed of the user'"'"'s eye movement, the speed of the user'"'"'s head movement, or a combination thereof; and

decrease resolution of the second area of the first digital image,

where the decrease in resolution of the second area of the first digital image is proportional to the speed of the user'"'"'s eye movement,

where the resolutions of the first area and the second area are modified substantially simultaneously, and

where the changes in resolution mimic a high-resolution viewing field and a low-resolution viewing field naturally experienced by the user when viewing real-world content.

8. The head-mounted device of claim 7, wherein the optical display surface is one of a plurality of optical display surfaces onto which the first digital image is projected.

9. The head-mounted device of claim 8, wherein each projection of the first digital image is modified identically upon determining where the focal point is located.

10. The head-mounted device of claim 9, wherein the first sensor is one of a plurality of sensors configured to continuously track speed and direction of the user'"'"'s eye movement to help identify the focal point of the user'"'"'s gaze.

11. The head-mounted device of claim 9, wherein the specific instructions further prompt the processor to:

identify a candidate area of a second digital image that is likely to become the subject of the user'"'"'s gaze at a second point in time using a predictive algorithm,

wherein the second digital image is displayed by the head-mounted device on the optical display surface, and

wherein the candidate area is part of the second area of the first digital image at the first point in time.

12. The head-mounted device of claim 11, wherein the predictive algorithm considers as input the direction of the user'"'"'s head movement, the speed of the user'"'"'s head movement, or both.

13. The head-mounted device of claim 11, wherein the predictive algorithm considers as input a digital element of the second digital image that is displayed at the second point in time and that is determined to likely become the subject of the user'"'"'s gaze.

14. The head-mounted device of claim 13, wherein the digital element is a character or an object, and wherein the first and second digital images are presented as part of a game or a video.

15. The head-mounted device of claim 7, wherein resolution of the first area is highest near the focal point of the user'"'"'s gaze.

16. The head-mounted device of claim 7, wherein determining where the focal point is located comprises:

recognizing a particular gesture that is performed by the user and that specifies where the focal point is located.

17. The head-mounted device of claim 16, wherein the particular gesture includes the user pointing at an object indicative of the focal point.

18. The head-mounted device of claim 7, wherein a size of the first area is related to speed of the user'"'"'s eye movement, speed of the user'"'"'s head movement, total time spent focusing on the focal point by the user, or some combination thereof.

19. A method for regionally modifying a resolution of digital content that is presented to a user of a head-mounted device to create a viewing experience that more accurately mimics the user'"'"'s natural high-resolution and low-resolution fields of view, the method comprising:

causing a digital image to be displayed by a head-mounted device;

determining where a focal point of a user'"'"'s gaze is located at a first point in time, the focal point being somewhere within the bounds of the digital image;

identifying a first area of the digital image that includes the focal point of the user'"'"'s gaze;

increasing resolution of the first area of the digital image;

identifying a second area of the digital image that is determined to likely become the focal point of the user'"'"'s gaze at a second point in time using a predictive algorithm;

monitoring a speed of the user'"'"'s eye movement and head movement;

determining that the focal point of the user'"'"'s gaze has begun to move toward the second area of the digital image;

increasing resolution of the second area of the digital image;

identifying a third area of the digital image located between the first area and the second idea of the digital image; and

decreasing resolution of the third area of the digital image,

where the resolutions of the first area, the second area, and the third area are modified substantially simultaneously as the focal point moves toward the second area of the digital image,

wherein a size of the first area and a size of the second area are based on the speed of the user'"'"'s eye movement, the speed of the user'"'"'s head movement, or a combination thereof, and

where the decrease in resolution of the third area of the digital image is proportional to the speed of the user'"'"'s eye movement, the speed of the user'"'"'s head movement, or both.

20. The method of claim 19, wherein the resolutions of the first area and the second area are highest near the focal point of the user'"'"'s gaze.

21. The method of claim 19, further comprising:

identifying a fourth area of the digital image composed of the remainder of the digital image not included in the first, second, or third areas of the digital image; and

decreasing resolution of the fourth area of the digital image.

22. The method of claim 21, wherein the third area has a different resolution than the fourth area.

23. The method of claim 19, wherein the second area of the digital image is located in a fixed, predetermined location.

24. The method of claim 19, wherein the second area of the digital image includes one or more permanent features that are always or nearly always part of the digital image and viewable by the user.

25. The method of claim 24, wherein the one or more permanent features includes an element of a user interface.

26. The method of claim 19, wherein the first area, second area, and third area are entirely distinct from one another.

27. The method of claim 19, wherein the decrease in resolution of the third area of the digital image is proportional to the speed of the user'"'"'s eye movement.