Rendering composite content on a head-mounted display including a high resolution inset

• US 10,373,294 B1
• Filed: 04/08/2018
• Issued: 08/06/2019
• Est. Priority Date: 10/04/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Wilson, Austin, Lanman, Douglas Robert, Trail, Nicholas Daniel, McEldowney, Scott Charles, McNally, Stephen James, Sulai, Yusufu Njoni Bamaxam
• Primary Examiner(s): Craddock, Robert J
• Application Number:
  US15/947,846
• Time in Patent Office: 485 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 3/4092 Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A head-mounted display (HMD) divides an image into a high resolution (HR) inset portion at a first resolution, a peripheral portion, and a transitional portion. The peripheral portion is downsampled to a second resolution that is less than the first resolution. The transitional portion is blended such that there is a smooth change in resolution that corresponds to a change in resolution between a fovea region and a non-fovea region of a retina. An inset region is generated using the HR inset portion and the blended transitional portion, and a background region is generated using the downsampled peripheral portion. The inset region is provided to a HR inset display, and the background region is provided to a peripheral display. An optics block combines the displayed inset region with the displayed background region to generate composite content.

Claims

1. A method comprising:

dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, wherein the HR inset portion is at a first resolution, and the transitional portion has an inner boundary between the transitional portion and the HR inset portion and an outer boundary between the transitional portion and the peripheral portion, wherein the peripheral portion is at a second resolution that is lower than the first resolution;

blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution;

generating an inset region using the HR inset portion of the image and the blended transitional portion of the image; and

providing, for display, the inset region and a background region to a display.

2. The method of claim 1, wherein dividing the image into the HR inset portion, the peripheral portion, and the transitional portion between the HR inset portion and the peripheral portion, further comprises:

applying a first fading function to the image that causes the inset portion and the transitional portion to fade to black, the faded image being the peripheral portion; and

applying a second fading function to some of the transitional portion of the image that surrounds the HR inset.

3. The method of claim 1, wherein dividing the image into the HR inset portion, the peripheral portion, and the transitional portion between the HR inset portion and the peripheral portion is based in part on an eye orientation of a user of a head mounted display (HMD).

4. The method of claim 1, wherein blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution, comprises:

applying a blurring function that blurs a portion of the transitional portion of the image.

5. The method of claim 1, wherein the background region is displayed using the display, and an optical anti-aliasing filter configured to optically blur some of the background region.

6. The method of claim 1, wherein the background region passes through an optical anti-aliasing filter before being combined with the inset portion that is displayed using the HR inset display, the optical anti-aliasing filter optically blurring some of the background region.

7. The method of claim 1, further comprising:

calibrating color for the inset portion and the transitional portion of the display such that the display is within a threshold range of color values.

8. The method of claim 1, further comprising:

calibrating luminance for the display such that it is within a threshold range of luminance values.

9. The method of claim 1 further including an optics block configured to combine the inset region and the background region.

10. A non-transitory computer-readable storage medium storing executable computer program instructions, the instructions executable to perform steps comprising:

dividing an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, wherein the HR inset portion is at a first resolution, and the transitional portion has an inner boundary between the transitional portion and the HR inset portion and an outer boundary between the transitional portion and the peripheral portion, wherein the peripheral portion is at a second resolution that is lower than the first resolution;

blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution;

generating an inset region using the HR inset portion of the image and the blended transitional portion of the image; and

providing, for display, the inset region and a background region to a display.

11. The computer readable medium of claim 10, wherein dividing the image into the HR inset portion, the peripheral portion, and the transitional portion between the HR inset portion and the peripheral portion, further comprises:

applying a first fading function to the image that causes the inset portion and the transitional portion to fade to black, the faded image being the peripheral portion; and

applying a second fading function to some of the transitional portion of the image that surrounds the HR inset.

12. The computer readable medium of claim 10, wherein dividing the image into the HR inset portion, the peripheral portion, and the transitional portion between the HR inset portion and the peripheral portion is based in part on an eye orientation of a user of a head mounted display (HMD).

13. The computer readable medium of claim 10, wherein blending the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution, comprises:

applying a blurring function that blurs a portion of the transitional portion of the image.

14. The computer readable medium of claim 10, wherein the background region is displayed using the display, and an optical anti-aliasing filter configured to optically blur some of the background region.

15. The computer readable medium of claim 10, wherein the background region passes through an optical anti-aliasing filter before being combined with the inset portion that is displayed using the HR inset display, the optical anti-aliasing filter optically blurring some of the background region.

16. The computer readable medium of claim 10, further comprising:

calibrating color for the inset portion and the transitional portion of the display such that the display is within a threshold range of color values.

17. The computer readable medium of claim 10, further comprising:

calibrating luminance for the display such that it is within a threshold range of luminance values.

18. The computer readable medium of claim 10, further including an optics block configured to combine the inset region and the background region.

19. A head-mounted display (HMD) comprising:

a controller configured to;

divide an image into a high resolution (HR) inset portion, a peripheral portion, and a transitional portion between the HR inset portion and the peripheral portion, wherein the HR inset portion is at a first resolution, and the transitional portion has an inner boundary between the transitional portion and the HR inset portion and an outer boundary between the transitional portion and the peripheral portion, wherein the peripheral portion is at a second resolution that is lower than the first resolution,

blend the transitional portion of the image such that there is a smooth change in resolution from the inner boundary at the first resolution to the outer boundary at the second resolution,

generate an inset region using the HR inset portion of the image and the blended transitional portion of the image, and

provide, for display, the inset region and a background region to a display; and

an optics block configured to;

combine the inset region and the background region to create composite content, wherein the inset region is inset into the background region, and

direct the composite content to an exit pupil of the HMD corresponding to a location of an eye of a user of the HMD.

20. The HMD of claim 19, wherein the controller is further configured to:

apply a first fading function to the image that causes the inset portion and the transitional portion to fade to black, the faded image being the peripheral portion; and

apply a second fading function to some of the transitional portion of the image that surrounds the HR inset.