Dynamic display calibration based on eye-tracking

• US 10,571,251 B2
• Filed: 02/26/2019
• Issued: 02/25/2020
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Edwin, Lionel E., Yeoh, Ivan L., Miller, Samuel A.
• Primary Examiner(s): Crawford, Jacinta M
• Application Number:
  US16/286,049
• Publication Number:
  US 20190226830A1
• Time in Patent Office: 364 Days
• Field of Search: 345 7, 345 8
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G09G 2320/0233 Improving the luminance or brightness uniformity across the screen
• CPC Class Codes: G09G 2320/0242 Compensation of deficiencies in the appearance of colours
• CPC Class Codes: G09G 2320/0276 for the purpose of adaptation to the characteristics of a display device, i.e. gamma correction
• CPC Class Codes: G09G 2320/028 by changing the viewing angle properties, e.g. widening the viewing angle, adapting the viewing angle to the view direction
• CPC Class Codes: G09G 2320/029 by monitoring one or more pixels in the display panel, e.g. by monitoring a fixed reference pixel
• CPC Class Codes: G09G 2320/0626 : for control of overall brightness
• CPC Class Codes: G09G 2320/0666 : for control of colour parameters, e.g. colour temperature
• CPC Class Codes: G09G 2320/0673 : for control of gamma adjustment, e.g. selecting another gamma curve
• CPC Class Codes: G09G 2320/0693 : Calibration of display systems
• CPC Class Codes: G09G 2340/0464 : Positioning
• CPC Class Codes: G09G 2360/16 : Calculation or use of calculated indices related to luminance levels in display data
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G09G 3/006 : Electronic inspection or testing of displays and display drivers, e.g. of LED or LCD displays testing individual LED's G01R31/2635; testing lamps G01R31/44; testing of optical features of LCD displays G02F1/1309
• CPC Class Codes: G09G 3/2092 : Details of a display terminals using a flat panel, the details relating to the control arrangement of the display terminal and to the interfaces thereto suitable for both CRT and flat panel G09G5/003; specific for a CRT G09G1/165
• CPC Class Codes: G09G 3/3406 : Control of illumination source illumination devices structurally associated with liquid crystal cells G02F1/1336
• CPC Class Codes: G09G 5/02 : characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: H04N 13/144 : for flicker reduction
• CPC Class Codes: H04N 13/327 : Calibration thereof
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/395 : with depth sampling, i.e. the volume being constructed from a stack or sequence of 2D image planes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof

Abstract

Examples of a light field metrology system for use with a display are disclosed. The light field metrology may capture images of a projected light field, and determine focus depths (or lateral focus positions) for various regions of the light field using the captured images. The determined focus depths (or lateral positions) may then be compared with intended focus depths (or lateral positions), to quantify the imperfections of the display. Based on the measured imperfections, an appropriate error correction may be performed on the light field to correct for the measured imperfections. The display can be an optical display element in a head mounted display, for example, an optical display element capable of generating multiple depth planes or a light field display.

Claims

1. A head-mounted display system comprising:

an eye-tracking camera;

a wearable display configured to be disposed in front of an eye of a wearer of the head-mounted display system, the display configured to present virtual image content to the eye of the wearer, the display having a plurality of depth planes associated therewith, wherein the display is configured to present different virtual images at different depth planes;

non-transitory data storage configured to store a calibration for the display, the calibration associated with a vector field comprising vectors corresponding to deviations between projected positions and expected positions of points of the virtual images at each of the depth planes; and

a hardware processor in communication with the eye-tracking camera, the wearable display, and the non-transitory data storage, the hardware processor programmed to;

determine, based on information from the eye-tracking camera, an eye position, relative to the display, of the wearer of the display;

access the calibration;

calculate, based at least partly on the calibration and the eye position, a correction to apply to at least one of the depth planes of the display to at least partially correct for an imperfection in the display; and

apply the correction to the at least one of the depth planes of the display.

2. The head-mounted display system of claim 1, wherein each of the plurality of depth planes is associated with a color and a depth relative to the display.

3. The head-mounted display system of claim 1, wherein the calibration is associated with a divergence or a curl of the vector field.

4. The head-mounted display system of claim 1, wherein the calibration corrects for a centration correction, a rotation correction, a scaling correction, a spatial mapping, a warping error, or an out-of-plane focal depth error for the at least one of the depth planes of the display.

5. The head-mounted display system of claim 1, wherein the wearable display comprises a first display associated with a first eye of the wearer and a second display associated with a second eye of the wearer, and wherein the calibration comprises a first calibration for the first display and a second calibration for the second display.

6. The head-mounted display system of claim 5, wherein the first calibration and the second calibration are the same and represent an average calibration for the first display and the second display.

7. The head-mounted display system of claim 1, wherein the wearable display comprises a light field display.

8. The head-mounted display system of claim 1, wherein the wearable display comprises a stackable waveguide assembly comprising a plurality of waveguides, and a depth plane in the plurality of depth planes is associated with at least one of the plurality of waveguides.

9. The head-mounted display system of claim 1, wherein the calibration further comprises a chromatic calibration that corrects for a chromatic imperfection of the wearable display that comprises a chromatic mismatch between two or more depth planes of the plurality of depth planes.

10. The head-mounted display system of claim 9, wherein the chromatic calibration comprises an adjustment in luminance between the two or more depth planes of the plurality of depth planes.

11. The head-mounted display system of claim 10, wherein the adjustment comprises a lowering of luminance values of the two or more depth planes to a minimum of the luminance values of the two or more depth planes.

12. The head-mounted display system of claim 1, wherein the calibration comprises a plurality of calibrations, and wherein each calibration in the plurality of calibrations is associated with a calibration position relative to the wearable display.

13. The head-mounted display system of claim 12, wherein to calculate the correction, the hardware processor is programmed to interpolate or to extrapolate among one or more of the plurality of calibrations, based at least in part on the calibration positions of the one or more of the plurality of calibrations and the determined eye position.

14. The head-mounted display system of claim 12, wherein to calculate the correction, the hardware processor is programmed to select from the plurality of calibrations a calibration having a calibration position that is closest to the determined eye position.

15. The head-mounted display system of claim 12, wherein the calibration positions are arranged in a grid across the wearable display.

16. The head-mounted display system of claim 1, wherein the hardware processor is programmed to apply the correction via a feedback loop that monitors change in the eye position.

17. The head-mounted display system of claim 1, wherein the hardware processor is programmed to determine a change in the eye position relative to a previous eye position, and to calculate the correction if the change exceeds a threshold.

18. The head-mounted display system of claim 1, wherein the vector field comprises a three-dimensional (3D) vector field corresponding to in-plane and out-of-plane deviations between projected positions and expected positions of points of the virtual images.

19. The head-mounted display system of claim 18, wherein the correction comprises an in-plane correction and an out-of-plane correction.

20. The head-mounted display system of claim 19, wherein the hardware processor is programmed to apply the in-plane correction before applying the out-of-plane correction.