Dynamic distortion correction for optical compensation

• US 10,551,879 B1
• Filed: 12/27/2018
• Issued: 02/04/2020
• Est. Priority Date: 06/01/2017
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Richards, Evan M., Lewis, Brant Carlton, Yoon, Youngshik, Shen, Shizhe
• Primary Examiner(s): Schnurr, John R
• Application Number:
  US16/234,284
• Time in Patent Office: 404 Days
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1637 Details related to the display arrangement, including those related to the mounting of the display in the housing
• CPC Class Codes: G06F 1/1675 Miscellaneous details related to the relative movement between the different enclosures or enclosure parts
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A distortion calibration system generates a distortion map for a head-mounted display (HMD). The system includes a camera that takes pictures of a test pattern displayed by the HMD. The images are taken at different camera positions and/or states of the HMD. The system determines a distortion map using the captured images, and uploads it to the HMD as part of, e.g., an optical model. The HMD comprises an electronic display, an eye tracking unit, an optics block, and a module. The module estimates distortion values based on eye tracking information from the eye tracking unit, an optical model, and a state of the HMD (e., a distance between the optics block and the electronic display). The module determines an adjusted state of the HMD using the estimated distortion values, the eye tracking information, and the optical model, and adjusts the state of the HMD to the adjusted state.

Claims

1. A distortion calibration system comprising:

a test mount configured to receive a head mounted display (HMD) under test, the test mount configured to hold the HMD under test as it presents a test pattern in accordance with calibration instructions, the HMD including an electronic display and an optics block;

a camera assembly including one or more cameras, the camera assembly configured to capture, for different positions of the one or more cameras, one or more images of the test pattern presented by the electronic display through the optics block in accordance with the calibration instructions; and

a calibration workstation configured to;

generate the calibration instructions,

provide the calibration instructions to the HMD under test and the camera assembly,

calculate distortion values for each of the different camera positions using the captured images,

generate a distortion map using the calculated distortion values and the camera positions, and

provide the distortion map to the HMD under test.

2. The distortion calibration system of claim 1, wherein the HMD under test includes a plurality of states, and each state is associated with a different distance between an optical element in the optics block and the electronic display, and the calibration instructions instruct the camera assembly to capture one or more images of the test pattern through the optics block of the HMD under test while the HMD under test is in a first state, of the plurality of states.

3. The distortion calibration system of claim 2, wherein the calibration instructions instruct the HMD under to test to be in a second state, of the plurality of states, and instruct the camera assembly to capture images of the test pattern through the optics block of the HMD under test while the HMD under test is in the second state.

4. The distortion calibration system of claim 3, where the HMD under test changes from the first state to the second state by adjusting a position of an optical element in the optics block relative to the electronic display.

5. The distortion calibration system of claim 3, where the HMD under test changes from the first state to the second state by adjusting a position of the electronic display relative to an optical element in the optics block.

6. The distortion calibration system of claim 1, wherein the calibration workstation is further configured to calculate distortion values for different positions of one or more cameras of the camera assembly, the electronic display, and an optical element in the optics block using the captured images.

7. The distortion calibration system of claim 1, wherein the HMD under test includes a plurality of states, and each state is associated with a different distance between an optical element in the optics block and the electronic display, and the distortion map is a mapping of values for one or more types of distortion to different camera positions for each of the plurality of states, wherein the different camera positions correspond to potential locations of an eye of a user of the HMD under test.

8. The distortion calibration system of claim 7, wherein the calibration workstation is further configured to generate the distortion map by:

creating distortion map entries associating each different camera position with the distortion values calculated for the camera position.

9. The distortion calibration system of claim 7, wherein the types of distortion are selected from a group consisting of:

field curvature, astigmatism, barrel distortion, pincushion distortion, mustache distortion, chromatic aberration, and chromatic aberration.

10. A method comprising:

generating calibration instructions for a head mounted display (HMD) and a camera assembly, the HMD including an electronic display and an optics block, and the camera assembly including one or more cameras;

providing, to the HMD, for presentation by the electronic display, a test pattern in accordance with the calibration instructions, the HMD held on a test mount configured to hold the HMD under test as it presents the test pattern;

providing, to the camera assembly, the calibration instructions, the calibration instructions instructing the camera assembly to capture, for different positions of the one or more cameras, one or more images of the test pattern presented by the electronic display through the optics block in accordance with the calibration instructions;

calculating distortion values for each of the different camera positions using the captured images;

generating a distortion map using the calculated distortion values and the camera positions; and

providing the distortion map to the HMD under test.

11. The method of claim 10, wherein the HMD under test includes a plurality of states, and each state is associated with a different image plane for the electronic display, and the calibration instructions instruct the camera assembly to capture one or more images of the test pattern through the optics block of the HMD under test while the HMD under test is in a first state, of the plurality of states.

12. The method of claim 11, wherein the calibration instructions instruct the HMD under to test to be in a second state, of the plurality of states, and instruct the camera assembly to capture images of the test pattern through the optics block of the HMD under test while the HMD under test is in the second state.

13. The method of claim 12, where the HMD under test changes from the first state to the second state by adjusting a position of an optical element in the optics block relative to the electronic display.

14. The method of claim 12, where the HMD under test changes from the first state to the second state by adjusting a position of the electronic display relative to an optical element in the optics block.

15. The method of claim 10, further comprising calculating distortion values for different positions of one or more cameras of the camera assembly, the electronic display, and an optical element in the optics block using the captured images.

16. The method of claim 10, wherein the HMD under test includes a plurality of states, and each state is associated with a different image plane for the electronic display, and the distortion map is a mapping of values for one or more types of distortion to different camera positions for each of the plurality of states, wherein the different camera positions correspond to potential locations of an eye of a user of the HMD under test.

17. The method of claim 16, wherein generating the distortion map comprises creating distortion map entries associating each different camera position with the distortion values calculated for the camera position.

18. The method of claim 16, wherein the types of distortion are selected from a group consisting of:

field curvature, astigmatism, barrel distortion, pincushion distortion, mustache distortion, chromatic aberration, and chromatic aberration.

19. A non-transitory computer-readable storage medium storing executable computer program instructions, the instructions executable to perform steps comprising:

generating calibration instructions for a head mounted display (HMD) and a camera assembly, the HMD including an electronic display and an optics block, and the camera assembly including one or more cameras;

providing, to the HMD, for presentation by the electronic display, a test pattern in accordance with the calibration instructions, the HMD held on a test mount configured to hold the HMD under test as it presents the test pattern;

providing, to the camera assembly, the calibration instructions, the calibration instructions instructing the camera assembly to capture, for different positions of the one or more cameras, one or more images of the test pattern presented by the electronic display through the optics block in accordance with the calibration instructions;

calculating distortion values for each of the different camera positions using the captured images;

generating a distortion map using the calculated distortion values and the camera positions; and

providing the distortion map to the HMD under test.

20. The non-transitory computer-readable storage medium of claim 19, wherein the HMD under test includes a plurality of states, and each state is associated with a different image plane for the electronic display, and the calibration instructions instruct the camera assembly to capture one or more images of the test pattern through the optics block of the HMD under test while the HMD under test is in a first state, of the plurality of states.