DEPTH BASED FOVEATED RENDERING FOR DISPLAY SYSTEMS

• US 20180275410A1
• Filed: 03/21/2018
• Published: 09/27/2018
• Est. Priority Date: 03/22/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Yeoh, Ivan Li Chuen, Edwin, Lionel Ernest, Samec, Nicole Elizabeth, Robaina, Nastasja U., Mathur, Vaibhav, Dalrymple, Tim, Schaefer, Jason, Carlisle, Clinton, Cheng, Hui-Chuan, Oh, Chulwoo, Premysler, Philip, Zhang, Xiaoyang, Carlson, Adam
• Granted Patent: US 11,644,669 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0185 Displaying image at variable distance
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/36 Level of detail
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/395 with depth sampling, i.e. the volume being constructed from a stack or sequence of 2D image planes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/002 Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices

Abstract

Methods and systems for depth-based foveated rendering in the display system are disclosed. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. Some embodiments include monitoring eye orientations of a user of a display system based on detected sensor information. A fixation point is determined based on the eye orientations, the fixation point representing a three-dimensional location with respect to a field of view. Location information of virtual objects to present is obtained, with the location information indicating three-dimensional positions of the virtual objects. Resolutions of at least one virtual object is adjusted based on a proximity of the at least one virtual object to the fixation point. The virtual objects are presented to a user by display system with the at least one virtual object being rendered according to the adjusted resolution.

Claims

1. A system comprising one or more processors and one or more computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

monitoring, based on information detected via one or more sensors, eye movements of a user;

determining, based on the eye movements, a fixation point at which the user'"'"'s eyes are fixating, wherein the fixation point is a three-dimensional location in a field of view of the user;

obtaining location information associated with one or more virtual objects to present to the user, the location information indicating three-dimensional positions of the virtual objects;

adjusting resolutions of at least one virtual object based, at least in part, on a proximity of the at least one virtual object to the fixation point; and

causing presentation, via a display device to the user, of the one or more virtual objects, the at least one virtual object being rendered according to the respective adjusted resolution.

2. The system of claim 1, wherein adjusting resolution of a virtual object comprises reducing a presentation quality of the virtual object.

3. The system of claim 2, wherein reducing a presentation quality of the virtual object comprises one or more of reducing a polygon count of the virtual object, adjusting primitives utilized to generate the virtual object, adjusting operations performed on the virtual object, adjusting texture information, adjusting color resolution or depth, and adjusting a number of rendering cycles or a frame rate.

4. The system of claim 1, wherein the field of view corresponds to a display frustum of the display device.

5. The system of claim 1, wherein the field of view comprises a plurality of zones, each zone representing a volume of space within the field of view in which virtual objects can be presented by the display device,

wherein the zones extend from an initial depth within the field of view to an end depth within the field of view, and

wherein the proximity of the one or more virtual objects from the fixation point corresponds to a distance separating a first zone in which the virtual object is to be presented from a second zone in which the fixation point is located.

6. The system of claim 5, wherein adjusting resolutions of the at least one virtual object comprises reducing resolution from a resolution initially assigned to the at least one virtual object, wherein the proximity corresponds to a total number of zones separating the first zone from the second zone.

7. The system of claim 5, wherein the second zone in which the fixation point is located is assigned a maximum resolution, and wherein remaining zones are assigned resolutions respectively reduced from the maximum resolution according to one or more drop-off rates.

8. The system of claim 5, wherein the operations further comprise:

providing a user retinal cone density profile,

wherein a drop-off rate on an individual depth plane substantially conforms to a drop-off rate of cone density in the user retinal cone density profile.

9. The system of claim 5, wherein the zones have three-dimensional polygonal shapes.

10. The system of claim 5, wherein the zones are concentric ellipsoids.

11. The system of claim 5, wherein a shape of the zones varies with distance to the second zone.

12. The system of claim 1, further comprising the display device, the display device comprising:

a plurality of stacked waveguides forming a display area and providing a view of an ambient environment through the display area, wherein at least some waveguides of the plurality of waveguides are configured to output light with different wavefront divergence that other waveguides.

13. The system of claim 1, wherein, for a first virtual object to be presented further in depth from the fixation point and which is determined to be within a threshold angular distance of a gaze of the user, the operations further comprise:

adjusting the resolution of the first virtual object based on a proximity of the first virtual object to the fixation point; and

causing the first virtual object to be blurred during presentation of the first virtual object to the user.

14. The system of claim 13, wherein causing the first virtual object to be blurred comprises:

performing a convolution of a Gaussian blur kernel with the first virtual object rendered according to the adjusted resolution; and

presenting, via the display device, the result of the convolution to the user.

15. The system of claim 1, wherein the one or more sensors comprise one or more of infrared sensors, ultraviolet sensors, and visible wavelength light imaging devices configured to detect gaze directions of eyes of the user.

129. The system of claim 1, wherein the operations further comprise:

determining an amount of light reaching a retina of an eye of the user; and

adjusting resolution of the at least one virtual object based on the amount of light reaching the retina.

130. The system of claim 1, wherein causing presentation of the one or more virtual objects comprises:

presenting temporally multiplexed first and second image streams to the user,

wherein virtual objects formed by the first image stream have a first pixel density, and wherein virtual objects formed by the second image stream have a second pixel density greater than the first pixel density, and

wherein virtual objects formed by the second image stream overlie corresponding portions of a field of view provided by the first image stream.

16. A display system comprising:

a display device configured to present virtual content to a user;

one or more processors; and

one or more computer storage media storing instructions that when executed by the system, cause the system to perform operations comprising;

monitoring information associated with eye movements of the user;

determining, based on the monitored information, a fixation point within a display frustum of the display device, the fixation point indicating a three-dimensional location being fixated upon by eyes of the user; and

based on the determined fixation point, presenting virtual content at three-dimensional locations within the display frustum, the virtual content being adjusted in resolution based on a proximity of the virtual content from the fixation point.

17. The system of claim 16, wherein the operations further comprise:

providing, by the display device to an outside system comprising one or more processors, the determined fixation point; and

receiving, by the display device and from the outside system, rendered virtual content for presentation, the virtual content being rendered at a resolution based on a respective proximity of the virtual content from the fixation point.

18. The system of claim 16, wherein adjusting resolution of particular virtual content based on a proximity from the fixation point comprises:

determining a reduction in a maximum resolution based on the proximity, the reduction being based on a drop-off rate; and

adjusting resolution of the particular virtual content based on the determined reduction.

19. The system of claim 16, wherein the display device comprises a plurality of waveguides, each waveguide presenting virtual content associated with respective depths based on three-dimensional locations of the virtual content.

20. The system of claim 16, wherein the display device presents virtual content based on foveated imaging with respect to the fixation point, the foveated imaging incorporating depth information associated with (1) the fixation point, and (2) virtual content.

21. A method comprising:

by a system of one or more processors,

monitoring, based on information detected via one or more sensors, eye orientations of a user of display device;

determining, based on the eye orientations, a fixation point at which the user'"'"'s eyes are fixating, the fixation point representing a three-dimensional location with respect to a field of view of the user;

obtaining location information associated with one or more virtual objects to present to the user, the location information indicating three-dimensional positions of the virtual objects;

adjusting resolutions of at least one virtual object based, at least in part, on a respective proximity from the at least one virtual object to the fixation point; and

causing presentation, via the display device to the user, of the one or more virtual objects, the at least one virtual object being rendered according to the respective adjusted resolution.

22. The method of claim 21, wherein adjusting resolution of a virtual object comprises reducing a presentation quality of the virtual object relative to a maximum possible presentation quality.

23. The method of claim 22, wherein reducing a presentation quality of the virtual object comprises reducing a polygon count of the virtual object, reducing primitives associated with the virtual object, or reducing texture information associated with the virtual object.

24. The method of claim 21, wherein the fixation point is determined for each frame presented to the user via the display device.

25. The method of claim 21, wherein the field of view comprises zones, each zone representing a volume of space within the field of view in which virtual objects can be presented;

wherein the zones extend, at least, from an initial depth within the field of view to an end depth within the field of view; and

wherein a proximity of a virtual object from the fixation point is determined based on a first zone in which the virtual object is to be presented relative to a second zone in which the fixation point is located.

26. The method of claim 21, wherein sensors comprise one or more of infrared sensors, ultraviolet sensors, and visible wavelength light imaging devices.

27-128. -128. (canceled)