Realtime lens aberration correction from eye tracking

• US 10,228,562 B2
• Filed: 08/25/2015
• Issued: 03/12/2019
• Est. Priority Date: 02/21/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Larsen, Eric
• Primary Examiner(s): Bibbins, LaTanya
• Application Number:
  US14/835,558
• Publication Number:
  US 20160091720A1
• Time in Patent Office: 1,295 Days
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0068 having means for controlling the degree of correction, e.g. using phase modulators, movable elements controlling the phase of light using moving or deformable elements G02B26/06
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features

Abstract

Systems and methods include receiving an image for presenting on a display screen of a head mounted display (HMD). The image is provided by an application. The received image is pre-distorted to enable optics provided in a HMD to render the image. An alignment offset is identified for an eye of a user wearing the HMD by determining a position of the eye relative to an optical axis of at least one lens of the optics of the HMD. The pre-distorted image provided by the application is adjusted to define a corrected pre-distorted image that accounts for the alignment offset. The corrected pre-distorted image is forwarded to the display screen of the HMD for rendering, such that the image presented through the optics of the HMD removes aberrations caused by the alignment offset.

Claims

1. A method, comprising:

receiving an image for presenting on a display screen of a head mounted display (HMD), the image being provided by an application, wherein the image is pre-distorted to enable optics provided in the HMD to render the image;

identifying an alignment offset of an eye of a user wearing the HMD by determining a position of the eye relative to an optical axis of at least one lens of the optics of the HMD;

determining a specific distortion correction for the alignment offset using one or more of pre-defined distortion maps configured for a shape and type of lens used in the optics, the pre-defined distortion maps identified for the eye position provide mapping that correlate different alignment offsets to corresponding locations within a visible field of view of the display screen of the HMD, the specific distortion correction for the alignment offset computed using linear interpolation of the different alignment offsets from the select ones of pre-defined distortion maps;

adjusting the pre-distorted image provided by the application by applying the specific distortion correction to define a corrected pre-distorted image that accounts for the alignment offset; and

sending the corrected pre-distorted image for rendering on the display screen of the HMD, such that the image presented through the optics removes aberration caused by the alignment offset,

wherein the method operations are performed by a processor.

2. The method of claim 1, wherein the alignment offset is identified for both eyes of a user.

3. The method of claim 1, wherein the optics of the HMD is configured to provide near-eye focus.

4. The method of claim 1, wherein identifying alignment offset includes identifying one of an x-shift, a y-shift, a z-shift, an x-and-y shift, an x-and-z shift, a y-and-z shift or an x-y-z shift.

5. The method of claim 1, wherein the position of the eye is captured using one or more internal sensors disposed on the HMD.

6. The method of claim 1, wherein the lens of the optics is one of a spherical lens, an aspherical lens, a holographic waveguide lens or any combinations thereof.

7. The method of claim 1, wherein identifying the alignment offset and adjusting the pre-distorted image are performed by a processor disposed within the HMD, the alignment offset is identified by tracking the position of the eye in substantial real-time as the image is being presented on the display screen of the HMD.

8. The method of claim 1, wherein identifying the alignment offset is performed by a processor of the HMD by tracking the position of the eye in substantial real-time as the image is being presented on the display screen of the HMD and adjusting the pre-distorted image is performed by a processor of a computing device that is communicatively connected to the HMD.

9. The method of claim 1, wherein the pre-defined distortion maps are two dimensional maps that are defined for each type of lens of the optics used in the HMD and for each primary color spectrum used for presenting the image and the selection of the pre-defined distortion maps include pre-defined distortion maps for the specific optics and for each color of the primary color spectrum that needs to be adjusted.

10. The method of claim 1, wherein each of the pre-defined distortion maps is associated with a particular quadrant of the visible field of view or an area covering center of the visible field of view.

11. The method of claim 1, wherein the one or more pre-defined distortion maps are generated based on optical simulation for each type and shape of lens of the optics used in the HMD.

12. A method, comprising:

a) identifying an alignment offset of an eye of a user wearing a head mounted display (HMD) by determining a position of the eye relative to an optical axis of at least one lens of optics provided in the HMD;

b) adjusting a pre-distorted image of media content provided by an application for rendering on a display screen of the HMD, wherein an amount of adjusting for the alignment offset is determined using one or more pre-defined distortion maps, the adjusting results in generation of a corrected, pre-distorted image that accounts for the alignment offset; and

c) sending the corrected, pre-distorted image for rendering on the display screen of the HMD, such that the image presented through the optics removes aberration caused by the alignment offset,

wherein the method operations are performed by a processor and wherein the method operations a)-c) are performed during rendering of frames of the media content included in the image at a frame rate at which the media content is being rendered at the HMD.

13. The method of claim 12, wherein the corrected pre-distorted image is presented to the display screen of the HMD for viewing by both eyes of a user.

14. The method of claim 12, wherein the optics of the HMD is configured to provide near-eye focus of the corrected pre-distorted image in front of both eyes of the user.

15. The method of claim 12, wherein identifying the alignment offset includes,

tracking the position of the eye in substantial real-time using one or more internal sensors disposed on the HMD;

determining shape and type of lens used in the optics of the HMD;

determining a shift in the position of the eye in relation to the optical axis of the lens of the optics of the HMD;

selecting the one or more pre-defined distortion maps based on the eye position, wherein the distortion maps provide a mapping of different locations within a visible field of view of the display screen to corresponding displacement offsets, the selection determined using a triangulation technique; and

computing the alignment offset using linear interpolation of the displacement offsets from the selected one or more pre-defined maps.

16. The method of claim 12, wherein identifying the alignment offset and adjusting the pre-distorted image is performed by a processor disposed within the HMD.

17. The method of claim 12, wherein identifying the alignment offset and adjusting the pre-distorted image is performed by a processor of a computing device that is communicatively connected to the HMD.

18. A head mounted display device, comprising:

a lens of optics for viewing a pre-distorted image of multimedia content rendered on a display screen of the head mounted display (HMD);

an eye position sensor configured to,

(a) determine position of an eye of a user wearing the HMD relative to an optical axis of the lens of optics provided in the HMD;

(b) determine alignment offset of the eye of the user; and

an optics distortion processor configured to,

adjust the pre-distorted image of media content provided by an application for rendering on the display screen of the HMD, wherein an amount of adjusting for the alignment offset is determined using one or more pre-defined distortion maps, the adjustment results in generation of a corrected, pre-distorted image that accounts for the alignment offset, wherein the corrected, pre-distorted image of the media content is transmitted to the display screen of the HMD for rendering,

wherein the optics distortion processor is configured to perform adjustment to the pre-distorted image of media content during rendering of frames of the media content at a frame rate at which the media content is being rendered at the HMD.

19. The device of claim 18, wherein the eye position sensor further includes a position tracker to track the position of the eye of the user as the image is being presented on the display screen of the HMD so as to perform adjustment to the pre-distorted image in substantial real-time based on information provided by the position tracker.

20. The device of claim 18, wherein the pre-distorted image is provided by a computing device executing the application, wherein the computing device is communicatively connected to the HMD.