AUTOMATIC VARIABLE VIRTUAL FOCUS FOR AUGMENTED REALITY DISPLAYS

• US 20160161740A1
• Filed: 02/12/2016
• Published: 06/09/2016
• Est. Priority Date: 11/08/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bar-Zeev, Avi, Lewis, John
• Granted Patent: US 9,588,341 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0159 with mechanical means other than scaning means for positioning the whole image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/0149 characterised by mechanical features G02B27/0176 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The technology provides an augmented reality display system for displaying a virtual object to be in focus when viewed by a user. In one embodiment, the focal region of the user is tracked, and a virtual object within the user focal region is displayed to appear in the focal region. As the user changes focus between virtual objects, they appear to naturally move in and out of focus as real objects in a physical environment would. The change of focus for the virtual object images is caused by changing a focal region of light processing elements in an optical path of a microdisplay assembly of the augmented reality display system. In some embodiments, a range of focal regions are swept through at a sweep rate by adjusting the elements in the optical path of the microdisplay assembly.

Claims

1. A method for generating an augmented reality display providing variable focus of virtual objects comprising:

determining a three dimensional field of view of a user viewing a scene including one or more real objects and one or more virtual objects in an augmented reality display image using a near-eye display device;

identifying a three dimensional location of one or more virtual objects within the field of view of the user;

determining a three dimensional current focal region of the user within the field of view;

identifying which of the one or more virtual objects are in the current focal region of the user;

placing the one or more virtual objects in the current focal region of the user in the augmented display image by adjusting the focal region of a microdisplay assembly of the near-eye display device; and

displaying an image including at least one virtual object .

2. The method of claim 1, wherein:

the determining the three dimensional field of view of the user viewing a scene further comprises;

determining a three-dimensional location of any object in the scene based on a three-dimensional model of the scene, and

determining the position and orientation of the user'"'"'s head within the three-dimensional model of the scene.

3. The method of claim 1, wherein:

the current focal region of the user is the Panum'"'"'s fusional area of the user'"'"'s eyes.

4. The method of claim 1, further comprising:

applying at least one artificial depth of field technique to virtual objects which are within the field of view of the user but outside the current focal region of the user as a function of distance from the current focal region.

5. The method of claim 1,wherein:

the placing the one or more virtual objects in the current focal region of the user in the augmented display image by adjusting the focal region of a microdisplay assembly of the near-eye display device comprises adjusting the displacement in an optical path of the microdisplay assembly.

6. The method of claim 5, wherein:

the microdisplay assembly comprises at least one optical element and a microdisplay unit aligned in the optical path to a reflecting element; and

the adjusting the displacement in the optical path of the microdisplay assembly further comprises

determining a displacement value, S₁, between a front nodal point of the at least one optical element and a target location of at least one virtual object, the at least one optical element being located in an optical path between a microdisplay and a reflecting element,

determining a displacement value, S₂, between the reflecting element and a rear nodal point of the at least one optical element for a focal length of the at least one lens, f, based on the equation 1/S₁+1/S₂=1/f; and

moving the at least one optical element to generate the determined displacement values Si and S2.

7. An augmented reality system providing variable focus of virtual objects comprising:

a see-through display unit comprising;

a microdisplay assembly for generating images for a display of the see-through display unit, the microdisplay assembly includes a reflecting element, at least one optical element, a microdisplay unit, and a variable virtual focus adjuster, the at least one optical element and the microdisplay unit being aligned in an optical path with the reflecting element, the variable virtual adjuster changing the focal region of the microdisplay assembly to place one or more virtual objects in a current focal region of a user,

control circuitry including a processor communicatively coupled to the microdisplay assembly, the control circuitry including a driver to control the variable virtual adjuster, the processor under the control of software directing the microdisplay unit to include at least one virtual object in a current user focal region, and

a display optically coupled to the microdisplay assembly for projecting three dimensional virtual images directed to at least one eye location.

8. The augmented reality system of claim 7 further comprising:

an armature for supporting at least one element of the microdisplay assembly and the armature being under the control of the variable virtual adjuster.

9. The augmented reality system of claim 7, wherein the variable virtual adjuster is a piezoelectric actuator.

10. The augmented reality system of claim 7, wherein:

the control circuitry further comprises a timing generator for providing a rate to the driver controlling the variable virtual adjuster;

the variable virtual adjuster sweeping at the rate through a range of displacements in the optical path of the microdisplay assembly, the range of displacements corresponding to a range of focal regions;

the range of focal regions including the current user focal region; and

the processor under the control of software directing the microdisplay unit to generate an image including any virtual object identified to be located in one of the focal regions of the range when the adjuster is at the displacement for that focal region.

11. The augmented reality system of claim 7, wherein:

the control circuitry further comprises a timing generator for providing a rate to the driver controlling the variable virtual adjuster;

the variable virtual adjuster sweeping at the rate through a range of polarization values of the at least one optical element in the optical path of the microdisplay assembly to change the focal length of the at least one optical element in the optical path, the range of polarization values corresponding to a range of focal regions;

the range of focal regions including the current user focal region; and

the processor under the control of software directing the microdisplay unit to generate an image including any virtual object identified to be located in one of the focal regions of the range when the adjuster is at the polarization for that focal region.

12. The augmented reality system of claim 7, wherein:

the control circuitry further comprises a timing generator for providing a rate to the driver controlling the variable virtual adjuster;

the at least one optical element in the optical path including a fluid lens;

the variable virtual adjuster sweeping at the rate through a range of radius of curvature values of the at least one fluid lens in the optical path of the microdisplay assembly to change the focal length of the at least one fluid lens, the range of radius of curvature values corresponding to a range of focal regions;

the range of focal regions including the current user focal region; and

the processor under the control of software directing the microdisplay unit to generate an image including any virtual object identified to be located in one of the focal regions of the range when the adjuster is at the radius of curvature for that focal region.

13. The augmented reality system of claim 10, wherein the display projects the images generated by the range of displacements to the at least one eye location.

14. The augmented reality system of claim 10, wherein the microdisplay generates a composite image including in-focus portions of the different images generated by the range of displacements, and

the display projects the composite image to the at least one eye location.

15. The augmented reality system of claim 10, wherein:

the see-through display unit being a head mounted display unit including a camera for capturing image data of one or more real objects in front of the user; and

one or more depth cameras for capturing three dimensional image data for positions of real objects in an environment surrounding the user;

a memory accessible to the processor for storing image data from the depth cameras and the camera on the head mounted display; and

the processor, under the control of software, generating a three- dimensional model of an augmented reality scene based on locations of one or more real objects derived from the image data captured by the cameras and locations of one or more virtual objects based on an executing software application.

16. The augmented reality system of claim 15, wherein:

the head mounted display unit including an orientation sensor and an eye tracking unit;

the processor under the control of software determining the field of view of the user based on the three dimensional model, and data from the orientation sensor and eye position data from the eye tracking unit;

identifying target locations of any virtual object within the user field of view;

determining which focal region any target location of any virtual object is within; and

selecting the range of focal regions based on the current user focal region and a motion trajectory of any virtual object within the user field of view controlled by the executing application.

17. One or more computer readable storage media having encoded thereon instructions for causing a processor to perform a method for generating virtual objects in different focal regions in an augmented reality display, the method comprising:

determining a three dimensional field of view of a user viewing a scene including real and virtual objects using a near-eye display device;

identifying at least one three dimensional focal region of one or more virtual objects within the field of view of the user;

during a sweep rate period, adjusting of a focal region of light processing elements of a microdisplay assembly for each focal region in a range of focal regions; and

generating an image at the at least one three dimensional focal region including one or more virtual objects in the user field of view.

18. The one or more computer readable storage media of claim 17, wherein the method further comprises displaying the generated image at the at least one three dimensional focal region during the sweep period.

19. The one or more computer readable storage media of claim 18, wherein at least one of the focal regions of the range is a current focal region of the user.

20. The one or more computer readable storage media of claim 18 wherein the method further comprises

displaying a composite multi-focus image including an in-focus portion of each generated image.