Rendering composite content on a head-mounted display including a high resolution inset

• US 9,779,478 B1
• Filed: 10/04/2016
• Issued: 10/03/2017
• Est. Priority Date: 10/04/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Oculus VR, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Wilson, Austin, Lanman, Douglas Robert, Trail, Nicholas Daniel, McEldowney, Scott Charles, McNally, Stephen James, Sulai, Yusufu Njoni Bamaxam
• Primary Examiner(s): Craddock, Robert
• Application Number:
  US15/284,898
• Time in Patent Office: 364 Days
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 3/4092 Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A head-mounted display (HMD) divides an image into a high resolution (HR) inset portion at a first resolution, a peripheral portion, and a transitional portion. The peripheral portion is downsampled to a second resolution that is less than the first resolution. The transitional portion is blended such that there is a smooth change in resolution that corresponds to a change in resolution between a fovea region and a non-fovea region of a retina. An inset region is generated using the HR inset portion and the blended transitional portion, and a background region is generated using the downsampled peripheral portion. The inset region is provided to a HR inset display, and the background region is provided to a peripheral display. An optics block combines the displayed inset region with the displayed background region to generate composite content.

Claims

1. A method comprising:

dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, wherein the image is at a first resolution corresponding to a resolution of a fovea region of a human eye, and the transitional portion has an inner boundary between the transitional portion and the HR inset portion and an outer boundary between the transitional portion and the peripheral portion;

downsampling the peripheral portion to a second resolution that is less than the first resolution, the second resolution corresponding to a resolution of a non-fovea region of the human eye;

blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution, the smooth change corresponding to a change in resolution between the fovea region and the non-fovea region;

generating an inset region using the HR inset portion of the image and the blended transitional portion of the image;

generating a background region using the downsampled peripheral portion;

providing, for display, the inset region to a HR inset display on a head-mounted display (HMD);

providing, for display, the background region to a peripheral display on the HMD, and

wherein an optics block combines the displayed inset region with the displayed background region to generate composite content.

2. The method of claim 1, wherein dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, further comprises:

applying a first fading function to the image that causes the inset portion and the transitional portion to fade to black, the faded image being the peripheral portion; and

applying a second fading function to some of the transitional portion of the image that surrounds the HR inset.

3. The method of claim 1, wherein dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion is based in part on an eye orientation of a user of the HMD.

4. The method of claim 1, wherein blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution, the smooth change corresponding to a change in resolution between the fovea region and the non-fovea region, comprises:

applying a blurring function that blurs a portion of the transitional portion of the image.

5. The method of claim 1, wherein the background region that is displayed using the peripheral display passes through an optical anti-aliasing filter before being combined with the inset portion that is displayed using the HR inset display, the optical anti-aliasing filter optically blurring some of the background region.

6. The method of claim 1, further comprising:

calibrating color for the peripheral display and the inset display such that they both are within a threshold range of color values.

7. The method of claim 1, further comprising:

calibrating luminance for the peripheral display and the inset display such that they both are within a threshold range of luminance values.

8. The method of claim 1, wherein the composite image has variable resolution that matches a resolution of a retina of a human eye.

9. A non-transitory computer-readable storage medium storing executable computer program instructions, the instructions executable to perform steps comprising:

dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, wherein the image is at a first resolution corresponding to a resolution of a fovea region of a human eye, and the transitional portion has an inner boundary between the transitional portion and the HR inset portion and an outer boundary between the transitional portion and the peripheral portion;

downsampling the peripheral portion to a second resolution that is less than the first resolution, the second resolution corresponding to a resolution of a non-fovea region of the human eye;

blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution, the smooth change corresponding to a change in resolution between the fovea region and the non-fovea region;

generating an inset region using the HR inset portion of the image and the blended transitional portion of the image;

generating a background region using the downsampled peripheral portion;

providing, for display, the inset region to a HR inset display on a head-mounted display (HMD);

providing, for display, the background region to a peripheral display on the HMD, and

wherein an optics block combines the displayed inset region with the displayed background region to generate composite content.

10. The computer readable medium of claim 9, wherein dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, further comprises:

applying a first fading function to the image that causes the inset portion and the transitional portion to fade to black, the faded image being the peripheral portion; and

applying a second fading function to some of the transitional portion of the image that surrounds the HR inset.

11. The computer readable medium of claim 9, wherein dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion is based in part on an eye orientation of a user of the HMD.

12. The computer readable medium of claim 9, wherein blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution, the smooth change corresponding to a change in resolution between the fovea region and the non-fovea region, comprises:

applying a blurring function that blurs a portion of the transitional portion of the image.

13. The computer readable medium of claim 9, wherein the background region that is displayed using the peripheral display passes through an optical anti-aliasing filter before being combined with the inset portion that is displayed using the HR inset display, the optical anti-aliasing filter optically blurring some of the background region.

14. The computer readable medium of claim 9, further comprising:

calibrating color for the peripheral display and the inset display such that they both are within a threshold range of color values.

15. The computer readable medium of claim 9, further comprising:

calibrating luminance for the peripheral display and the inset display such that they both are within a threshold range of luminance values.

16. The computer readable medium of claim 9, wherein the composite image has variable resolution that matches a resolution of a retina of a human eye.

17. A head-mounted display (HMD) comprising:

a controller configured to;

divide an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, wherein the image is at a first resolution corresponding to a resolution of a fovea region of a human eye, and the transitional portion has an inner boundary between the transitional portion and the HR inset portion and an outer boundary between the transitional region and the peripheral portion,

downsample the peripheral portion to the second resolution, the second resolution corresponding to a resolution of a non-fovea region of the human eye,

blend the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution, the smooth change corresponding to a change in resolution between the fovea region and a non-fovea region of the human eye,

generate an inset region using the HR inset portion of the image and the blended transitional portion of the image,

generate a background region using the downsampled peripheral portion; and

a high resolution (HR) inset display that is configured to display the inset region;

a peripheral display that is configured to display the background region;

an optics block configured to;

combine the inset region and the background region to create composite content, wherein the inset region is inset into the background region, and

direct the composite content to an exit pupil of the HMD corresponding to a location of an eye of a user of the HMD.

18. The HMD of claim 17, wherein the controller is further configured to:

apply a first fading function to the image that causes the inset portion and the transitional portion to fade to black, the faded image being the peripheral portion; and

apply a second fading function to some of the transitional portion of the image that surrounds the HR inset.

19. The HMD of claim 17, further comprising:

an eye tracking unit configured to monitor an orientation of an eye of a user of the HMD, and

wherein the controller is configured to divide an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion based in part on the orientation of the eye.

20. The HMD of claim 17, wherein the controller is further configured to:

apply a blurring function that blurs a portion of the transitional portion of the image.

21. The HMD of claim 17, further comprising:

an optical anti-aliasing filter that optically blurs a portion of the background content.

22. The HMD of claim 17, wherein the composite image has variable resolution that matches a resolution of a retina of a human eye.