Automatic variable virtual focus for augmented reality displays

• US 9,292,973 B2
• Filed: 11/08/2010
• Issued: 03/22/2016
• Est. Priority Date: 11/08/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bar-Zeev, Avi, Lewis, John
• Primary Examiner(s): Broome, Said
• Application Number:
  US12/941,825
• Publication Number:
  US 20120113092A1
• Time in Patent Office: 1,961 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0159 with mechanical means other than scaning means for positioning the whole image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/0149 characterised by mechanical features G02B27/0176 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The technology provides an augmented reality display system for displaying a virtual object to be in focus when viewed by a user. In one embodiment, the focal region of the user is tracked, and a virtual object within the user focal region is displayed to appear in the focal region. As the user changes focus between virtual objects, they appear to naturally move in and out of focus as real objects would in a physical environment. The change of focus for the virtual object images is caused by changing a focal region of light processing elements in an optical path of a microdisplay assembly of the augmented reality display system. In some embodiments, a range of focal regions are swept through at a sweep rate by adjusting the elements in the optical path of the microdisplay assembly.

Claims

1. A method for displaying virtual objects with variable focus by an augmented reality system comprising:

identifying by one or more processors a three dimensional location of one or more virtual objects within a user field of view of a near-eye display device of the augmented reality system, the near-eye display device comprising a microdisplay assembly and a display unit optically coupled to receive one or more images from the microdisplay assembly, the microdisplay assembly including one or more light processing elements including at least one optical element and a microdisplay unit aligned in an optical path with the at least one optical element;

determining by the one or more processors a three dimensional current focal region of a user wearing the near-eye display device within the user field of view;

identifying which of the one or more virtual objects are in the current focal region of the user based on the three dimensional locations of the one or more virtual objects within the user field of view;

adjusting a focal region of a the microdisplay assembly for generating image data in the current focal region by moving a position of the one or more light processing elements in the optical path of the microdisplay assembly under control of the one or more processors;

generating a series of layered image data at different focal regions, by the microdisplay assembly, for each of the one or more virtual objects, such that the one or more virtual objects identified as being in the current focal region at the adjusted focal region are generated in focus and the one or more virtual objects outside of the focal region are generated blurred; and

receiving from the microdisplay assembly and displaying by the display unit the generated image data of the one or more virtual objects.

2. The method of claim 1, wherein determining by the one or more processors a three dimensional current focal region of a user wearing the near-eye display device within the user field of view further comprises:

determining the three dimensional current focal region of the user based on a Panum'"'"'s fusional area.

3. The method of claim 1, further comprising:

applying at least one artificial depth of field technique to a virtual object which is within the user field of view but outside the current focal region of the user as a function of distance from the current focal region.

15. The method of claim 1, wherein moving the one or more light processing elements in the optical path of the microdisplay assembly under control of the one or more processors further comprises:

determining by the one or more processors a displacement value, S1, between a front nodal point of the at least one optical element and a target location of at least one virtual object, the at least one optical element being located in an optical path between a microdisplay and a reflecting element,

determining by the one or more processors a displacement value, S2, between the reflecting element and a rear nodal point of the at least one optical element for a focal length of the at least one optical element, f, based on the equation 1/S1+1/S2=1/f, and

moving the at least one optical element under control of the one or more processors to generate the determined displacement values S1 and S2.

4. An augmented reality system providing variable focus of virtual objects comprising:

a near-eye support structure;

one or more processors;

a see-through display unit positioned by the near-eye support structure to be seen through by a user eye;

a microdisplay assembly for generating one or more images, supported by the near-eye support structure and optically coupled to the see-through display unit for outputting image data for a current user focal region to the see-through display unit, the microdisplay assembly includes one or more light processing elements including a microdisplay, at least one optical element, and a variable virtual focus adjuster, the at least one optical element and the microdisplay being aligned in an optical path, the variable virtual adjuster changing the focal region of the microdisplay assembly to place one or more virtual objects in a current focal region of a user by moving a position of the one or more light processing elements in the optical path under control of the one or more processors;

an armature for supporting at least one light processing element of the microdisplay assembly and stabilizing movement of the at least one light processing element moving along the optical path;

the one or more processors controlling the microdisplay assembly for generating image data at the changed focal region of the one or more virtual objects in the current focal region; and

the display unit receiving the generated image data from the microdisplay assembly and displaying the generated image data including a driver to control the variable virtual adjuster, the processor under the control of software directing the microdisplay to include at least one virtual object in a current user focal region; and

a display optically coupled to the microdisplay assembly via the optical path for displaying three dimensional virtual images.

5. The augmented reality system of claim 4, wherein the armature is under the control of the variable virtual adjuster.

6. The augmented reality system of claim 4, wherein the variable virtual adjuster is a piezoelectric actuator.

7. The augmented reality system of claim 4, further comprising a timing generator, under control of the one or more processors, for providing a rate to a driver controlling the variable virtual adjuster;

the variable virtual adjuster moving at least one light processing element of the microdisplay assembly at the rate through a range of displacements in the optical path of the microdisplay assembly, the range of displacements corresponding to a range of focal regions;

the range of focal regions including a focal region at which image generated by the microdisplay results in image data being displayed by the display unit at the current user focal region; and

the one or more processors directing the microdisplay to generate an image including any virtual object identified to be located in a respective focal region of the range when the respective displacement in the range of displacements is reached.

8. The augmented reality system of claim 4, wherein:

the at least one optical element of the microdisplay assembly comprises a set of insertable lenses and the variable virtual adjuster comprises a respective arm for moving each insertable lens in the set into and out of the optical path; and

the variable virtual adjuster changing the focal region of the microdisplay assembly to place one or more virtual objects in a current focal region of a user by moving the one or more light processing elements in the optical path under control of one or more processors further comprises moving the respective arm for moving a position of a respective insertable lens in the set with respect to the optical path for changing the focal region.

9. The augmented reality system of claim 4, wherein:

the at least one optical element of the microdisplay assembly comprises a set of lenses in a rotatable support under control of the variable virtual adjuster; and

the variable virtual adjuster changing the focal region of the microdisplay assembly to place one or more virtual objects in a current focal region of a user by moving the one or more light processing elements in the optical path under control of one or more processors further comprises the variable virtual adjuster rotating the rotatable support to change a lens of the set of lenses in the optical path for changing the focal region.

10. The augmented reality system of claim 7, wherein the display unit projects the images generated by the range of displacements to the at least one eye location.

11. The augmented reality system of claim 7, wherein the microdisplay generates a composite image including in-focus portions of the different images generated by the range of displacements, and

the display unit projects the composite image to the at least one eye location.

12. The augmented reality system of claim 7 further comprising:

one or more cameras supported by the near-eye support structure for capturing image data and depth data of one or more real objects in front of the user; and

a memory accessible to the one or more processors for storing the image data and the depth data; and

the one or more processors, generating a three-dimensional model of an augmented reality scene based on locations of one or more real objects derived from the image data and the depth data captured by the cameras and locations of one or more virtual objects based on an executing software application.

13. The augmented reality system of claim 12 further comprising an orientation sensor and an eye tracking unit supported by the near-eye support structure;

the one or more processors determining the field of view of the user based on the three dimensional model, and data from the orientation sensor and eye position data from the eye tracking unit, identifying target locations of any virtual object within the user field of view, determining which focal region any target location of any virtual object is within, and

selecting the range of focal regions based on the current user focal region and a motion trajectory of any virtual object within the user field of view controlled by the executing application.

16. The augmented reality system of claim 4, wherein moving the one or more light processing elements in the optical path under control of the one or more processors further comprises:

determining by the one or more processors a displacement value, S1, between a front nodal point of the at least one optical element and a target location of at least one virtual object, the at least one optical element being located in an optical path between a microdisplay and a reflecting element,

determining by the one or more processors a displacement value, S2, between the reflecting element and a rear nodal point of the at least one optical element for a focal length of the at least one optical element, f, based on the equation 1/S1+1/S2=1/f, and

moving the at least one optical element under control of the one or more processors to generate the determined displacement values S1 and S2.

14. One or more computer readable non-volatile storage media having encoded thereon instructions for causing a processor to perform a method for displaying virtual objects with variable focus by an augmented reality system, the method comprising:

identifying by one or more processors a three dimensional location of one or more virtual objects within a user field of view of a near-eye display device of the augmented reality system, the near-eye display device comprising a microdisplay assembly and a display unit optically coupled to receive one or more images from the microdisplay assembly, the microdisplay assembly including one or more light processing elements including at least one optical element and a microdisplay unit aligned in an optical path with the at least one optical element, and the at least one optical element of the microdisplay assembly comprises a set of insertable lenses such that changing a lens of the set of lenses in the optical path changes the focal region;

determining by the one or more processors a three dimensional current focal region of a user wearing the near-eye display device within the user field of view;

identifying which of the one or more virtual objects are in the current focal region of the user based on the three dimensional locations of the one or more virtual objects within the user field of view;

adjusting a focal region of the microdisplay assembly for generating image data in the current focal region by moving a position of the one or more light processing elements in the optical path of the microdisplay assembly under control of the one or more processors;

generating image data, by the microdisplay assembly, of the one or more virtual objects identified as being in the current focal region at the adjusted focal region; and

receiving from the microdisplay assembly and displaying by the display unit the generated image data of the one or more virtual objects.

17. The one or more computer readable non-volatile storage media of claim 14, the method further comprising:

wherein moving the one or more light processing elements in the optical path of the microdisplay assembly under control of the one or more processors further comprises;

determining by the one or more processors a displacement value, S1, between a front nodal point of the at least one optical element and a target location of at least one virtual object, the at least one optical element being located in an optical path between a microdisplay and a reflecting element,

determining by the one or more processors a displacement value, S2, between the reflecting element and a rear nodal point of the at least one optical element for a focal length of the at least one optical element, f, based on the equation 1/S1+1/S2=1/f, and

moving the at least one optical element under control of the one or more processors to generate the determined displacement values S1 and S2.