Dynamic display calibration based on eye-tracking

• US 10,260,864 B2
• Filed: 11/02/2016
• Issued: 04/16/2019
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Edwin, Lionel E., Yeoh, Ivan L., Miller, Samuel A.
• Primary Examiner(s): Crawford, Jacinta M
• Application Number:
  US15/341,822
• Publication Number:
  US 20170124928A1
• Time in Patent Office: 895 Days
• Field of Search: 345 7, 345 8
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G09G 2320/0233 Improving the luminance or brightness uniformity across the screen
• CPC Class Codes: G09G 2320/0242 Compensation of deficiencies in the appearance of colours
• CPC Class Codes: G09G 2320/0276 for the purpose of adaptation to the characteristics of a display device, i.e. gamma correction
• CPC Class Codes: G09G 2320/028 by changing the viewing angle properties, e.g. widening the viewing angle, adapting the viewing angle to the view direction
• CPC Class Codes: G09G 2320/029 by monitoring one or more pixels in the display panel, e.g. by monitoring a fixed reference pixel
• CPC Class Codes: G09G 2320/0626 : for control of overall brightness
• CPC Class Codes: G09G 2320/0666 : for control of colour parameters, e.g. colour temperature
• CPC Class Codes: G09G 2320/0673 : for control of gamma adjustment, e.g. selecting another gamma curve
• CPC Class Codes: G09G 2320/0693 : Calibration of display systems
• CPC Class Codes: G09G 2340/0464 : Positioning
• CPC Class Codes: G09G 2360/16 : Calculation or use of calculated indices related to luminance levels in display data
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G09G 3/006 : Electronic inspection or testing of displays and display drivers, e.g. of LED or LCD displays testing individual LED's G01R31/2635; testing lamps G01R31/44; testing of optical features of LCD displays G02F1/1309
• CPC Class Codes: G09G 3/2092 : Details of a display terminals using a flat panel, the details relating to the control arrangement of the display terminal and to the interfaces thereto suitable for both CRT and flat panel G09G5/003; specific for a CRT G09G1/165
• CPC Class Codes: G09G 3/3406 : Control of illumination source illumination devices structurally associated with liquid crystal cells G02F1/1336
• CPC Class Codes: G09G 5/02 : characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: H04N 13/144 : for flicker reduction
• CPC Class Codes: H04N 13/327 : Calibration thereof
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/395 : with depth sampling, i.e. the volume being constructed from a stack or sequence of 2D image planes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof

Abstract

Examples of a light field metrology system for use with a display are disclosed. The light field metrology may capture images of a projected light field, and determine focus depths (or lateral focus positions) for various regions of the light field using the captured images. The determined focus depths (or lateral positions) may then be compared with intended focus depths (or lateral positions), to quantify the imperfections of the display. Based on the measured imperfections, an appropriate error correction may be performed on the light field to correct for the measured imperfections. The display can be an optical display element in a head mounted display, for example, an optical display element capable of generating multiple depth planes or a light field display.

Claims

1. A display system comprising:

an eye-tracking camera;

a wearable display comprising a plurality of display color layers, each display color layer of the plurality of display color layers configured to output a different color;

non-transitory data storage configured to store a plurality of chromatic calibrations for the display, each chromatic calibration in the plurality of chromatic calibrations associated with a calibration position relative to the display, wherein each chromatic calibration in the plurality of chromatic calibrations corrects for a chromatic imperfection of the display comprising a chromatic mismatch between two or more display color layers of the plurality of display color layers; and

a hardware processor in communication with the eye-tracking camera, the display, and the non-transitory data storage, the hardware processor programmed to;

determine, based on information from the eye-tracking camera, an eye position, relative to the display, of the user of the display;

access, based at least partly on the determined eye position, one or more of the plurality of chromatic calibrations;

calculate, based at least in part on the one or more of the plurality of chromatic calibrations, a correction to apply to at least one of the two or more display color layers of the display to at least partially correct the chromatic imperfection of the display; and

apply the correction to the at least one of the two or more display color layers of the display.

2. The display system of claim 1, wherein a number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more.

3. The display system of claim 1, wherein the calibration positions are distributed across the display in a grid.

4. The display system of claim 3, wherein the grid comprises a 2×

2, a 3×

3, a 5×

5, or a 9×

9grid.

5. The display system of claim 1, wherein the one or more of the plurality of chromatic calibrations comprises a chromatic calibration associated with a calibration position that is closest to the eye position.

6. The display system of claim 1, wherein to calculate the correction, the hardware processor is programmed to interpolate or to extrapolate among the one or more of the plurality of chromatic calibrations, based at least in part on the calibration positions of the one or more of the plurality of chromatic calibrations and the determined eye position.

7. The display system of claim 1, wherein the display comprises a first display associated with a first eye of the user and a second display associated with a second eye of the user, and the hardware processor is programmed to determine the eye position of the user relative to the first display and to apply the determined eye position for calculating the correction for the second display.

8. The display system of claim 1, wherein the display comprises a first display associated with a first eye of the user and a second display associated with a second eye of the user, and wherein at least some of the plurality of calibrations represent an average calibration for the first display and the second display.

9. The display system of claim 1, wherein the display comprises a light field display.

10. The display system of claim 1, wherein the display comprises a stackable waveguide assembly comprising a plurality of waveguides, and a display color layer in the plurality of display color layers comprises at least one of the plurality of waveguides.

11. The display system of claim 1, wherein the display is configured as a head-mounted wearable display system.

12. The display system of claim 1, wherein the hardware processor is further programmed to correct for one or more spatial imperfection that comprise one or more of an in-plane translation, rotation, scaling, or warping error or an out-of-plane or focal depth error.

13. The display system of claim 1, wherein each chromatic calibration in the plurality of chromatic calibrations comprises an adjustment in luminance between the two or more display color layers of the plurality of display color layers.

14. The display system of claim 13, wherein:

the two or more display color layers of the plurality of display color layers comprise a first color layer having a first luminance and a second color layer having a second luminance, the first luminance greater than the second luminance, and

the adjustment in luminance comprises a reduction in the first luminance to a value of the second luminance.

15. A method for calibrating a wearable display, the method comprising:

under control of a dynamic calibration system performed by computer hardware;

determining an eye position for a user of the display, the display comprising a plurality of display color layers, each display color layer of the plurality of display color layers configured to output a different color;

accessing, based at least partly on the determined eye position, a chromatic calibration for the display, wherein the chromatic calibration is selected based upon an associated calibration position and the determined eye position, and wherein the chromatic calibration is configured to correct for a chromatic imperfection of the display comprising a chromatic mismatch between two or more display color layers of the plurality of display color layers;

calculating, based at least in part on the accessed chromatic calibration, a correction to apply to at least one of the two or more display color layers of the display to at least partially correct for the chromatic imperfection in the display, wherein calculating the correction comprises calibrating for the chromatic imperfection of the display; and

applying the correction to the at least one of the two or more display color layers of the display.

16. The method of claim 15, wherein accessing the chromatic calibration comprises selecting one or more chromatic calibrations from a plurality of chromatic calibrations, wherein each chromatic calibration is associated with a different calibration position relative to the display.

17. The method of claim 16, wherein the calibration positions are arranged in a grid across the display.

18. The method of claim 16, wherein calculating the correction comprises interpolating or extrapolating among the one or more of the plurality of chromatic calibrations, based upon the associated calibration positions of the one or more of the plurality of chromatic calibrations and the determined eye position.

19. The method of claim 15, further comprising accessing an image of an eye of the user of the display, and determining the eye position based at least in part upon the image of the eye.

20. A wearable display system comprising:

an inward-facing imaging system;

a display comprising a plurality of display color layers, each display color layer of the plurality of display color layers configured to output a different color;

non-transitory data storage configured to store a plurality of chromatic calibrations for the display, each chromatic calibration in the plurality of chromatic calibrations associated with a calibration position relative to the display and configured to correct for a chromatic imperfection of the display comprising a chromatic mismatch between two or more display color layers of the plurality of display color layers; and

a hardware processor in communication with the inward-facing imaging system, the display, and the non-transitory data storage, the hardware processor programmed to;

determine, using the inward-facing imaging system, an eye position relative to the display of the user of the display;

calculate, based at least partly on the determined eye position and one or more of the plurality of chromatic calibrations, a correction to apply to at least one of the two or more display color layers of the display to at least partially correct for the chromatic imperfection in the display; and

apply the correction to the at least one of the two or more display color layers of the display.

21. The wearable display system of claim 20, wherein the hardware processor is programmed to apply the correction via a feedback loop that monitors change in the eye position.

22. The wearable display system of claim 20, wherein the hardware processor is programmed to determine a change in the eye position relative to a previous eye position, and to calculate the correction if the change exceeds a threshold.

23. The wearable display system of claim 20, wherein the hardware processor is further programmed to correct for a spatial imperfection that comprises one or more of an in-plane translation, rotation, scaling, or warping error or an out-of-plane or focal depth error.

24. The wearable display system of claim 20, wherein each chromatic calibration in the plurality of chromatic calibrations comprises an adjustment in luminance between the two or more display color layers of the plurality of display color layers.

25. The wearable display system of claim 24, wherein:

the two or more display color layers of the plurality of display color layers comprise a first color layer having a first luminance and a second color layer having a second luminance, the first luminance greater than the second luminance, and

the adjustment in luminance comprises a reduction in the first luminance to a value of the second luminance.