Accommodation based optical correction

• US 10,379,356 B2
• Filed: 04/07/2017
• Issued: 08/13/2019
• Est. Priority Date: 04/07/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Nicholls, William Aaron, Zannoli, Marina, Lanman, Douglas Robert, Trail, Nicholas Daniel
• Primary Examiner(s): Alexander, William R
• Application Number:
  US15/482,564
• Publication Number:
  US 20170293146A1
• Time in Patent Office: 858 Days
• Field of Search: 359630
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features

Abstract

An auto-focus head-mounted display (HMD) dynamically generates aberration-adjusted images based on measured accommodation of user'"'"'s eye(s). An aberration-adjusted image is an image distorted to correct aberrations that would otherwise occur at a retina of the user due to image light passing through optics of the HMD. The aberration-adjusted image corrects the aberrations of the HMD and “accounts” for the aberrations of the eye so that the resulting retinal image is free of optical aberrations due to the HMD but preserves correct eye optical aberrations that are correlated with a current accommodative state of the eye.

Claims

1. A head mounted display (HMD) comprising:

at least one processor;

an electronic display element configured to display a virtual scene to a user wearing the HMD;

an optics block configured to direct light from the electronic display element to an exit pupil of the HMD;

an eye tracking system configured to determine an eye position of each eye of the user; and

memory including instructions that, when executed by the at least one processor, cause the at least one processor to;

determine a position and orientation of the HMD worn by the user;

identify, for a frame of the virtual scene, an image for rendering based on the determined position and orientation of the HMD;

determine an accommodative state of a plurality of accommodative states for an eye of the user based on the eye position of each eye of the user;

determine optical aberration information for the determined eye position of each eye of the user for the determined accommodative state, the optical aberration information corresponding to one or more optical aberrations caused by the optics block as the light from the electronic display element passes through the optics block to the exit pupil, the optical aberration information being different for at least one eye position relative to other eye positions within at least one of the plurality of accommodative states;

determine a focus state for the optics block corresponding to a combination of the accommodative state and the determined eye position of the user, the determined focus state being mapped to a distortion correction of a plurality of distortion corrections, each distortion correction adding pre-distortion to the content for the determined focus state; and

display the identified image for the frame of the virtual scene on the electronic display element with the pre-distortion of the distortion correction for the determined focus state, the light of the image displayed with the pre-distortion being complementarily distorted by the optics block.

2. The HMD of claim 1, wherein determining the accommodation information for the eye of the user includes:

determining vergence of the user'"'"'s eyes based on the eye position of each eye of the user; and

triangulating the accommodation information based on the determined vergence and an interpupillary distance between each eye of the user.

3. The HMD of claim 1, wherein determining the accommodation information for the eye of the user includes:

illuminating each eye of the user with an infrared (IR) light source;

capturing one or more images of each eye of the user with an IR detector;

determining a shape of a lens of each eye of the user from the captured one or more images; and

comparing the shape of the lens of each eye to a model that maps shapes of eye lenses to accommodation states to determine the accommodation information.

4. The HMD of claim 1, wherein the optical aberration information is determined by:

estimating optical aberration measurements for the optics block for each of a range of accommodative states, each accommodative state being based at least in part on a different eye position; and

indexing the optical aberration measurements by eye position.

5. The HMD of claim 4, wherein the one or more optical aberrations include at least one of field curvature or chromatic aberration.

6. The HMD of claim 1, further comprising:

a varifocal actuation block configured to change a focal length of the optics block based at least in part on accommodation information and, wherein the instructions that, when executed by the at least one processor, further cause the at least one processor to;

adjust a focal length of the optics block based on the determined focus state using the varifocal actuation block.

7. The HMD of claim 6, wherein adjusting the focal length of the optics block includes:

changing a distance between the optics block and the electronic display element to provide accommodation for the determined focus state.

8. A method comprising:

determining, by a head-mounted display (HMD), a position and orientation of the HMD worn by a user viewing content displayed by the HMD, the HMD directing light from an electronic display element presenting the content to an exit pupil of the HMD via an optics block;

identifying, for a frame of the content, an image for rendering based on the determined position and orientation of the HMD;

determining, by the HMD, an eye position for each eye of the user;

determining an accommodative state of a plurality of accommodative states for an eye of the user based on the eye position for each eye of the user;

determining optical aberration information for the determined eye position of each eye of the user for the determined accommodative state, the optical aberration information corresponding to one or more optical aberrations caused by the optics block as light from the electronic display element passes through the optics block to the exit pupil, the optical aberration information being different for at least one eye position relative to other eye positions within at least one of the plurality of accommodative states;

determining a focus state for the optics block corresponding to a combination of the accommodative state and the determined eye position of the user, the determined focus state being mapped to a distortion correction of a plurality of distortion corrections, each distortion correction adding pre-distortion to the content for the determined focus state; and

displaying, based on the determined focus state, the identified image for the frame of the content on the electronic display element with the pre-distortion of the distortion correction, the light of the image displayed with the pre-distortion is complementarily distorted by the optics block to mitigate the one or more optical aberrations caused by the optics block for the determined eye position.

9. The method of claim 8, further comprising:

tracking each eye of the user to an updated position for an additional image of an additional frame of the content;

determining updated optical aberration information for the updated position of each eye of the user;

selecting an updated focus state for the updated position of each eye of the user relative to the exit pupil from the focus state; and

displaying the additional image for focusing to the exit pupil by the optics block after application of the updated focus state.

10. The method of claim 8, wherein determining the accommodation information for the eye of the user includes:

determining vergence of the user'"'"'s eyes based on the eye position of each eye of the user; and

triangulating the accommodation information based on the determined vergence and an interpupillary distance between each eye of the user.

11. The HMD of claim 8, wherein determining the accommodation information for the eye of the user includes:

illuminating each eye of the user with an infrared (IR) light source;

capturing one or more images of each eye of the user with an IR detector;

determining a shape of a lens of each eye of the user from the captured one or more images; and

comparing the shape of the lens of each eye to a model that maps shapes of eye lenses to accommodation states to determine the accommodation information.

12. The HMD of claim 8, wherein the optical aberration information is determined by:

estimating optical aberration measurements for the optics block for each of a range of accommodative states for one or more users, each accommodative state being based at least in part on a different eye position of the one or more users; and

indexing the optical aberration measurements by eye position.

13. The method of claim 8, further comprising:

adjusting a focal length of the optics block based on the determined focus state using a varifocal actuation block.

14. The method of claim 13, wherein adjusting the focal length of the optics block includes:

changing a distance between the optics block and the electronic display element to provide accommodation for the determined focus state.

15. The method of claim 13, wherein the varifocal actuation block is configured to change the focal length of the optics block by changing a shape or optical path length of a lens included in the optics block to provide accommodation for the determined focus state.

16. The method of claim 15, wherein changing the shape or optical path length of the lens of the optics block includes using at least one selected from a group consisting of:

a shape-changing polymer lens, a liquid lens and electrowetting, an Alvarez-Lohmann lens, a deformable membrane mirror, a liquid crystal (electroactive) lens, a phase-only spatial light modulator (SLM), and any combination thereof.

17. A head mounted display (HMD) comprising:

at least one processor;

an electronic display element configured to display a virtual scene to a user wearing the HMD;

an optics block configured to direct light from the electronic display element to an exit pupil of the HMD;

an eye tracking system including an image capturing element, the eye tracking system configured to determine an eye position of each eye of the user;

a varifocal actuation block configured to change a focal length of the optics block based at least in part on accommodation information; and

memory including instructions that, when executed by the at least one processor, cause the at least one processor to;

determine a position and orientation of the HMD worn by the user;

identify, for a frame of the virtual scene, an image for rendering based on the determined position and orientation of the HMD;

determine an accommodative state of a plurality of accommodative states for an eye of the user based on the eye position of each eye of the user;

determine optical aberration information for the determined eye position of each eye of the user for the determined accommodative state, the optical aberration information corresponding to one or more optical aberrations caused by the optics block as light from the electronic display element passes through the optics block to the exit pupil, the optical aberration information being different for at least one eye position relative to other eye positions within at least one of the plurality of accommodative states;

determine a focus state for the optics block corresponding to a combination of the accommodative state and the determined eye position of the user, the determined focus state being mapped to a pre-distortion correction of a plurality of pre-distortion corrections, each distortion correction adding pre-distortion to the content for the determined focus state;

adjust a focal length of the optics block based on the determined focus state using the varifocal actuation block; and

display the identified image for the frame of the virtual scene on the electronic display element with the pre-distortion of the distortion correction for the determined focus state, the light of the image displayed with the pre-distortion being complementarily distorted by the optics block to mitigate the one or more optical aberrations caused by the optics block for the determined eye position.

18. The HMD of claim 17, wherein determining the accommodation information for the eye of the user includes:

determining vergence of the user'"'"'s eyes based on the eye position of each eye of the user; and

triangulating the accommodation information based on the determined vergence and an interpupillary distance between each eye of the user.

19. The HMD of claim 17, wherein determining the accommodation information for the eye of the user includes:

illuminating each eye of the user with an infrared (IR) light source;

capturing one or more images of each eye of the user with an IR detector;

determining a shape of a lens of each eye of the user from the captured one or more images; and

comparing the shape of the lens of each eye to a model that maps shapes of eye lenses to accommodation states to determine the accommodation information.

20. The HMD of claim 17, wherein the optical aberration information is determined by:

estimating optical aberration measurements for the optics block for each of a range of accommodative states, each accommodative state being based at least in part on a different eye position; and

indexing the optical aberration measurements by eye position.