Virtual, augmented, and mixed reality systems and methods

• US 10,529,063 B2
• Filed: 08/22/2017
• Issued: 01/07/2020
• Est. Priority Date: 08/22/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Rodriguez, Jose Felix, Perez, Ricardo Martinez
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US15/683,677
• Publication Number:
  US 20180053284A1
• Time in Patent Office: 868 Days
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3243 Power saving in microcontroller unit
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G09G 2330/021 Power management, e.g. power saving
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A virtual, augmented, or mixed reality display system includes a display configured to display virtual, augmented, or mixed reality image data, the display including one or more optical components which introduce optical distortions or aberrations to the image data. The system also includes a display controller configured to provide the image data to the display. The display controller includes memory for storing optical distortion correction information, and one or more processing elements to at least partially correct the image data for the optical distortions or aberrations using the optical distortion correction information.

Claims

1. A virtual, augmented, or mixed reality display system comprising:

a display configured to display distortion-corrected image data, the display comprising one or more optical components which introduce optical distortions or aberrations to the distortion-corrected image data, wherein the display comprises a plurality of at least partially distinct optical paths; and

a display controller configured to provide the distortion-corrected image data to the display, the display controller comprising;

memory for storing optical distortion correction information, wherein the optical distortion correction information comprises separate optical distortion correction information for respective colors of the optical paths of the display, and

one or more processing elements to at least partially correct non-distortion-corrected image data for the optical distortions or aberrations using the optical distortion correction information;

wherein the display controller is configured to pre-distort the non-distortion-corrected image data to obtain the distortion-corrected image data by determining a distortion-corrected pixel at a first location (x, y) for the distortion-corrected image data based on one or more non-distortion-corrected pixels near a different second location (x′

, y′

) in the non-distortion-corrected image data received by the display controller; and

wherein the display controller is configured to use the optical distortion correction information to determine the second location (x′

, y′

) in the non-distortion-corrected image data.

2. The system of claim 1, wherein the optical distortion correction information is used by the display controller to pre-distort the non-distortion-corrected image data in a way that is at least partially inversely related to the optical distortions or aberrations.

3. The system of claim 1, wherein the (x′

, y′

) coordinates of the second location are fractional numbers.

4. The system of claim 3, wherein the display controller is configured to determine the distortion-corrected pixel at the first location (x, y) by interpolating between a plurality of non-distortion-corrected pixels surrounding the second location (x′

, y′

).

5. The system of claim 4, wherein the display controller uses bilinear interpolation.

6. The system of claim 1, wherein the optical distortion correction information is stored in the form of one or more lookup tables.

7. The system of claim 6, wherein the one or more lookup tables are stored in a compressed format, and wherein the display controller is configured to expand the one or more lookup tables before correcting for the optical distortions or aberrations using the optical distortion correction information.

8. The system of claim 1, wherein the optical distortion correction information further comprises information for performing one or more image warp operations, and wherein the display controller is further configured to perform the one or more image warp operations on the non-distortion-corrected image data.

9. The system of claim 1, wherein the display is configured to display the colors sequentially.

10. A method in a virtual, augmented, or mixed reality display system, the method comprising:

providing distortion-corrected image data to be shown on a display, the display comprising one or more optical components which introduce optical distortions or aberrations to the distortion-corrected image data, wherein the display comprises a plurality of at least partially distinct optical paths;

storing optical distortion correction information, wherein the optical distortion correction information comprises separate optical distortion correction information for respective colors of the optical paths of the display;

at least partially correcting non-distortion-corrected image data for the optical distortions or aberrations using the optical distortion correction information; and

displaying the distortion-corrected image data to the user with the display;

wherein the act of at least partially correcting the non-distortion-corrected image data comprises;

pre-distorting the non-distortion-corrected image data to obtain the distortion-corrected image data by determining a distortion-corrected pixel at a first location (x, y) for the distortion-corrected image data based on one or more non-distortion-corrected pixels near a different second location (x′

, y′

) in the non-distortion-corrected image data; and

using the optical distortion correction information to determine the second location (x′

, y′

) in the non-distortion-corrected image data.

11. The method of claim 10, further comprising using the optical distortion correction information to pre-distort the non-distortion-corrected image data in a way that is at least partially inversely related to the optical distortions or aberrations.

12. The method of claim 10, wherein the (x′

, y′

) coordinates of the second location are fractional numbers.

13. The method of claim 12, further comprising determining the distortion-corrected pixel at the first location (x, y) by interpolating between a plurality of non-distortion-corrected pixels surrounding the second location (x′

, y′

).

14. The method of claim 13, further comprising using bilinear interpolation.

15. The method of claim 10, further comprising storing the optical distortion correction information in the form of one or more lookup tables.

16. The method of claim 15, further comprising storing the one or more lookup tables in a compressed format, and expanding the one or more lookup tables before correcting for the optical distortions or aberrations using the optical distortion correction information.

17. The method of claim 10, wherein the optical distortion correction information further comprises information for performing one or more image warp operations, and further comprising performing the one or more image warp operations on the non-distortion-corrected image data.

18. The method of claim 10, wherein the display is configured to display the colors sequentially.