Focus adjustment method for a virtual reality headset

• US 10,241,569 B2
• Filed: 12/08/2015
• Issued: 03/26/2019
• Est. Priority Date: 12/08/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Lanman, Douglas Robert, Ebert, Ryan Michael, Fix, Alexander Jobe, Nicholls, William Aaron, Mounier, Michael Sean, Cavin, Robert Dale
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): Suteerawongsa, Jarurat
• Application Number:
  US14/963,126
• Publication Number:
  US 20170160798A1
• Time in Patent Office: 1,204 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0185 Displaying image at variable distance
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

A virtual reality headset displays a three-dimensional (3D) virtual scene and includes a varifocal element to dynamically adjust a focal length of an optics block included in the virtual reality headset based on a location in the virtual scene where the user is looking. The headset tracks a user'"'"'s eyes to approximate gaze lines and determines a plane of focus for a frame of the virtual scene as the intersection of the gaze lines. The varifocal element adjusts the focal length of the optics block so the optics block is focused at the plane of focus, which keeps the user'"'"'s eyes in a zone of comfort as vergence and accommodation change. Based on the plane of focus, the virtual reality headset may provide depth cues, such as depth of field blur, to planes in the virtual scene deeper in the user'"'"'s field of view than the plane of focus.

Claims

1. A method comprising:

determining, using at least one image capturing element inside a head-mounted display (HMD), an eye position for each eye of a user, the HMD including an optics block configured to focus light from a display presenting a virtual scene to the user and a varifocal element configured to change a focal length of the optics block based on a viewing location within the virtual scene of the user;

determining gaze lines for each eye of the user based at least in part on each eye position, the determined gaze lines identifying the viewing location within the virtual scene;

estimating a vergence depth for the viewing location of the user based on an estimated intersection of the gaze lines;

comparing the estimated vergence depth to a depth associated with the viewing location identified by scene geometry data of the virtual scene;

replacing the estimated vergence depth with the depth of the viewing location based on a difference between the estimated vergence depth and the depth of the viewing location being greater than a threshold;

adjusting the focal length of the optics block based on the depth of the viewing location identified by the scene geometry data to provide accommodation for the user using the varifocal element; and

displaying the virtual scene on the display of the HMD.

2. The method of claim 1, wherein displaying the virtual scene on the display of the HMD comprises:

determining, for a frame of the virtual scene, a state of the optics block based on a state of the optics block during presentation of one or more previously presented frames of the virtual scene, each state of the optics block associated with a distortion correction map that corrects for optical error introduced by the state of the optics block;

determining a distortion correction map associated with the determined state of the optics block for the frame of the virtual scene to correct optical error introduced by the determined state of the optics block for the frame;

applying the determined distortion correction map to the frame of the virtual scene; and

displaying the virtual scene after application of the determined distortion correction map on the display of the HMD.

3. The method of claim 2, further comprising:

determining, for the determined eye position of each eye, at least one of a distance between each eye and the optics block or a position of each eye relative to the optics block, each state of the optics block is associated with a focal length of the optics block and at least one of the distance between each eye and the optics block or the position of each eye relative to the optics block, each distortion correction map further correcting for optical error introduced by at least one of the distance between each eye and the optics block or the position of each eye relative to the optics block.

4. The method of claim 1, wherein displaying the identified objects in the virtual scene with synthetic blur comprises:

identifying objects in the virtual scene with distances from an eye of the user that are different than a distance of the depth of the viewing location identified by the scene geometry data from the eye of the user;

displaying the identified objects in the virtual scene with synthetic blur

determining a level of synthetic blur for an identified object based at least in part on a distance between the identified object and the depth of the viewing location identified by the scene geometry data; and

displaying the identified object in the virtual scene with the determined level of synthetic blur.

5. The method of claim 1, wherein adjusting the focal length of the optics block based on the depth of the viewing location identified by the scene geometry data to provide accommodation for the user using the varifocal element comprises:

changing a distance between the optics block and the display by moving the optics block.

6. The method of claim 1, wherein adjusting the focal length of the optics block based on the depth of the viewing location identified by the scene geometry data to provide accommodation for the user using the varifocal element comprises:

changing a shape or optical path length of a lens of the optics block.

7. The method of claim 6, wherein changing the shape or the optical path length of the lens of the optics block includes using at least one selected from a group consisting of a shape-changing polymer lens, a liquid lens and electrowetting, an Alvarez-Lohmann lens, a deformable membrane mirror, a liquid crystal (electroactive) lens, a phase-only spatial light modulator (SLM), and any combination thereof.

8. The method of claim 1, wherein determining, using at least one image capturing element inside the HMD, an eye position for each eye of the user comprises:

determining a position and an orientation of the HMD using one or more position sensors; and

determining the viewing location of the user based at least in part on the position and the orientation of the HMD.

9. A method comprising:

determining, using at least one image capturing element inside a head-mounted display (HMD), an eye position for each eye of a user, the HMD including an optics block configured to focus light from a display presenting a virtual scene to an exit pupil and a varifocal element configured to change a focal length of the optics block based on a viewing location within the virtual scene of the user;

determining a three-dimensional (3D) gaze point of the user based at least in part on the eye position determined for each eye, the determined three-dimensional gaze point identifying the viewing location within the virtual scene;

estimating a vergence depth for the viewing location of the user based on 3D gaze point, the estimated vergence depth corresponding to an object of focus for the user;

comparing the estimated vergence depth to a depth associated with the viewing location identified by scene geometry data for the object within the virtual scene;

replacing the estimated vergence depth with the depth associated with the viewing location based on a difference between the estimated vergence depth and the depth of the viewing location being greater than a threshold;

adjusting the focal length of the optics block for the virtual scene based on the depth of the viewing location identified by the scene geometry data using the varifocal element to provide accommodation for the object of focus; and

displaying, the virtual scene on the display of the HMD.

10. The method of claim 9, wherein adjusting the focal length of the optics block for the virtual scene based on the depth associated with the viewing location identified by the scene geometry data using the varifocal element comprises:

changing a distance between the optics block and the display by moving the optics block.

11. The method of claim 9, wherein adjusting the focal length of the optics block for the virtual scene based on the depth associated with the viewing location identified by the scene geometry data using the varifocal element comprises:

changing a shape or optical path length of a lens included in the optics block.

12. The method of claim 11, wherein changing the shape or optical path length of the lens included in the optics block comprises using at least one selected from a group consisting of a shape-changing polymer lens, a liquid lens and electrowetting, an Alvarez-Lohmann lens, a deformable membrane mirror, a liquid crystal (electroactive) lens, a phase-only spatial light modulator (SLM), and any combination thereof.

13. The method of claim 9, wherein displaying the virtual scene on the display of the HMD comprises:

identifying objects in the virtual scene with distances from an eye of the user other than a distance of the depth associated with the viewing location identified by the scene geometry data from the eye of the user; and

displaying the identified objects in the virtual scene with synthetic blur.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the processor to:

capture, using a camera of ahead-mounted display (HMD), image information corresponding to an eye position for each eye of a user, the HMD including an optics block configured to focus light from a display presenting a virtual scene to an exit pupil and a varifocal element configured to change a focal length of the optics block based on a viewing location of the user within the virtual scene;

determine gaze lines of the user based at least in part on each eye position, the determined gaze lines identifying the viewing location within the virtual scene;

estimate a vergence depth for the viewing location of the user based on an estimated intersection of the gaze lines;

compare the estimated vergence depth to a depth associated with the viewing location identified by the scene geometry data;

replace the estimated vergence depth with the depth associated with the viewing location based on a difference between the estimated vergence depth and the depth of the viewing location being greater than a threshold;

adjust the focal length of the optics block for a frame of the virtual scene using the varifocal element based on the depth associated with the viewing location identified by the scene geometry data to provide accommodation for the plane of focus; and

display the virtual scene on the display of the HMD.

15. The non-transitory computer-readable storage medium of claim 14, wherein displaying the virtual scene on the display of the HMD comprises:

determine, for a frame of the virtual scene, a state of the optics block based on a state of the optics during presentation of one or more previously presented frames of the virtual scene, each state of the optics block associated with a distortion correction map that corrects for optical error introduced by the state of the optics block;

determine a distortion correction map associated with the determined state of the optics block for the frame of the virtual scene to correct optical error introduced by the determined state of the optics block for the frame;

apply the determined distortion correction map to the frame of the virtual scene; and

display the virtual scene after application of the determined distortion correction map on the display of the HMD.

16. The non-transitory computer-readable storage medium of claim 14, wherein displaying the virtual scene on the display of the HMD comprises:

identify objects in the virtual scene with distances from an eye of the user other than a distance of the depth associated with the viewing location identified by the scene geometry data from the eye of the user; and

display the identified objects in the virtual scene with synthetic blur.

17. The non-transitory computer-readable storage medium of claim 14, wherein adjusting the focal length of the optics block based on the depth associated with the viewing location identified by the scene geometry data to provide accommodation for the user using the varifocal element comprises:

change a distance between the optics block and the display by moving the optics block.

18. The non-transitory computer-readable storage medium of claim 14, wherein adjusting the focal length of the optics block based on the depth associated with the viewing location identified by the scene geometry data to provide accommodation for the user using the varifocal element comprises:

change a shape or optical path length of a lens of the optics block.

19. The non-transitory computer-readable storage medium of claim 18, wherein change the shape or optical path length of the lens of the optics block includes using at least one selected from a group consisting of a shape-changing polymer lens, a liquid lens and electrowetting, an Alvarez-Lohmann lens, a deformable membrane mirror, a liquid crystal (electroactive) lens, a phase-only spatial light modulator (SLM), and any combination thereof.