Stabilization plane determination based on gaze location

• US 10,078,367 B2
• Filed: 03/09/2017
• Issued: 09/18/2018
• Est. Priority Date: 04/29/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Michail, Ashraf Ayman, Sylvan, Roger Sebastian Kevin, Miller, Quentin Simon Charles, Kipman, Alex Aben-Athar
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US15/454,715
• Publication Number:
  US 20170177082A1
• Time in Patent Office: 558 Days
• Field of Search: 345419
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09G 2310/0235 Field-sequential colour display
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2320/106 : Determination of movement vectors or equivalent parameters within the image
• CPC Class Codes: G09G 3/3266 : Details of drivers for scan electrodes
• CPC Class Codes: G09G 5/346 : for systems having a bit-mapped display memory

Abstract

Embodiments are described herein for determining a stabilization plane to reduce errors that occur when a homographic transformation is applied to a scene including 3D geometry and/or multiple non-coplanar planes. Such embodiments can be used, e.g., when displaying an image on a head mounted display (HMD) device, but are not limited thereto. In an embodiment, a rendered image is generated, a gaze location of a user is determined, and a stabilization plane, associated with a homographic transformation, is determined based on the determined gaze location. This can involve determining, based on the user'"'"'s gaze location, variables of the homographic transformation that define the stabilization plane. The homographic transformation is applied to the rendered image to thereby generate an updated image, and at least a portion of the updated image is then displayed.

Claims

1. A method for displaying an image on a display, the method comprising:

generating a rendered image of a scene including 3D geometry and/or multiple non-coplanar planes;

determining a gaze location of a user;

determining a stabilization plane, associated with a homographic transformation, based on the determined gaze location, wherein the stabilization plane is selected such that the stabilization plane reduces errors that occur when the homographic transform associated with the stabilization plane is applied to the rendered image of the scene including 3D geometry and/or multiple non-coplanar planes;

applying the homographic transformation to the rendered image to thereby generate an updated image of the scene including 3D geometry and/or multiple non-coplanar planes; and

displaying at least a portion of the updated image of the scene including 3D geometry and/or multiple non-coplanar planes on the display.

2. The method of claim 1, wherein the determined the stabilization plane comprises determining, based on the determined gaze location, variables of the homographic transformation that define the stabilization plane.

3. The method of claim 1, wherein the determining the stabilization plane comprises:

determining a depth of the gaze location; and

determining a depth of the stabilization plane as being equal to the depth of the gaze location.

4. The method of claim 1, wherein the determining the stabilization plane based on the determined gaze location of the user comprises:

identifying a virtual object that is at or within a range of the gaze location; and

determining the stabilization plane based on the identified virtual object.

5. The method of claim 4, wherein the determining the stabilization plane based on the identified virtual object comprises determining a stabilization plane depth and a stabilization plane orientation based on a depth and an orientation of the identified virtual object.

6. The method of claim 5, wherein the identified virtual object is a planar two-dimensional virtual object, and determining the stabilization plane depth and the stabilization plane orientation as being equal, respectively, to the depth and the orientation of the identified planar two-dimensional virtual object.

7. The method of claim 5, wherein the identified virtual object is an elongated three-dimensional virtual object, and determining the stabilization plane depth and the stabilization plane orientation as being equal, respectively, to a depth and an orientation of a plane that bisects a center of the identified elongated three-dimensional vertical object along its elongated length.

8. The method of claim 4, wherein the identified virtual object takes up at least a specified percentage of a field of view and includes a plurality of subparts, and the determining the stabilization plane based on the identified virtual object comprises determining the stabilization plane based on one of the subparts that is closest to the gaze location.

9. The method of claim 4, wherein the identified virtual object includes a plurality of subparts, and the determining the stabilization plane based on the identified virtual object comprises determined the stabilization plane based on two or more of the subparts by weighting each of the two or more subparts'"'"' influence on the stabilization plane in dependence on how close the subpart is to the gaze location.

10. The method of claim 1, wherein a plurality of virtual objects are being displayed, the determining the stabilization plane based on the determined gaze location of the user comprises determining the stabilization plane based on two or more of the virtual objects by weighting each of the two or more virtual objects'"'"' influence on the stabilization plane in dependence on how close the virtual object is to the gaze location.

11. The method of claim 1, wherein the determining the gaze location of the user comprises performing eye tracking to determine the gaze location of the user.

12. The method of claim 1, wherein the determining the gaze location of the user comprises:

performing head tracking to determine a head position of the user;

estimating the gaze location of the user based the determined head position; and

determining the gaze location as being the estimated gaze location.

13. A device, comprising:

one or more processors that generate a rendered image of a scene including 3D geometry and/or multiple non-coplanar planes, determine a stabilization plane associated with a homographic transformation based on a gaze location of a user, wherein the stabilization plane is selected such that the stabilization plane reduces errors that occur when the homographic transform associated with the stabilization plane is applied to the rendered image of the scene including 3D geometry and/or multiple non-coplanar planes, and apply the homographic transformation to the rendered image to thereby generate an updated image of the scene including 3D geometry and/or multiple non-coplanar planes; and

a display that is in communication with the one or more processors and displays at least a portion of the updated image of the scene including 3D geometry and/or multiple non-coplanar planes.

14. The device of claim 13, wherein the one or more processors determine, based on the gaze location, variables of the homographic transformation that define the stabilization plane.

15. The device of claim 13, further comprising:

an eye tracker that determines the gaze location of the user.

16. The device of claim 13, wherein the device comprises a head mounted display (HMD) device.

17. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors associated with a head mounted display (HMD) device to perform a method for displaying an image on a display of the HMD device, the method comprising:

generating a rendered image of a scene including 3D geometry and/or multiple non-coplanar planes;

determining a gaze location of a user;

determining a stabilization plane, associated with a homographic transformation, based on the determined gaze location, wherein the stabilization plane is selected such that the stabilization plane reduces errors that occur when the homographic transform associated with the stabilization plane is applied to the rendered image of the scene including 3D geometry and/or multiple non-coplanar planes;

applying the homographic transformation to the rendered image to thereby generate an updated image of the scene including 3D geometry and/or multiple non-coplanar planes; and

displaying at least a portion of the updated image of the scene including 3D geometry and/or multiple non-coplanar planes on the display.

18. The one or more processor readable storage devices of claim 17, wherein the determined the stabilization plane comprises determining, based on the determined gaze location, variables of the homographic transformation that define the stabilization plane.

19. The one or more processor readable storage devices of claim 17, wherein the determining the stabilization plane based on the determined gaze location of the user comprises:

identifying a virtual object that is at or within a range of the gaze location; and

determining the stabilization plane based on the identified virtual object.

20. The one or more processor readable storage devices of claim 19, wherein the determining the stabilization plane based on the identified virtual object comprises determining a stabilization plane depth and a stabilization plane orientation based on a depth and an orientation of the identified virtual object.