Automatic focus improvement for augmented reality displays

• US 9,304,319 B2
• Filed: 11/18/2010
• Issued: 04/05/2016
• Est. Priority Date: 11/18/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bar-Zeev, Avi, Lewis, John
• Primary Examiner(s): Giesy, Adam R
• Application Number:
  US12/949,650
• Publication Number:
  US 20120127062A1
• Time in Patent Office: 1,965 Days
• Field of Search: 345 4- 8
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 3/14 of variable focal length
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An augmented reality system provides improved focus of real and virtual objects. A see-through display device includes a variable focus lens a user looks through. A focal region adjustment unit automatically focuses the variable focus lens in a current user focal region. A microdisplay assembly attached to the see-through display device generates a virtual object for display in the user'"'"'s current focal region by adjusting its focal region. The variable focus lens may also be adjusted to provide one or more zoom features. Visual enhancement of an object may also be provided to improve a user'"'"'s perception of an object.

Claims

1. An augmented reality system providing improved focus of objects comprising:

a see-through display device having a support structure, the see-through display positioned by the support structure and one or more physical environment facing cameras positioned by the support structure for capturing depth images of a space in which the see-through display device is located;

a memory storing software and data;

a processor determining a three-dimensional current field of view as seen through the see-through display based on a three dimensional model of the space in which the see-through display device is located and the captured depth images under the control of software, the processor having access to the memory and being communicatively coupled to a microdisplay assembly;

the processor being operable to determine a current focal region in the three dimensional current field of view;

the microdisplay assembly attached to the see-through display device, the microdisplay assembly including a microdisplay, one or more optical elements, and a variable virtual focus adjuster for changing an optical path between the microdisplay and the one or more optical elements of the microdisplay assembly;

the processor causing the microdisplay assembly to generate an in-focus virtual image which will display in-focus in the determined current focal region by controlling the variable virtual focus adjuster to adjust a focal region of the microdisplay assembly based on detected edges of the objects to identify distinct objects for enhancement; and

the see-through display of the see-through display device being optically coupled to receive and to display the in-focus virtual image generated for the determined current focal region.

2. The system of claim 1, further comprising:

the see-through display includes a variable focus lens positioned to be seen through;

the processor receiving input requesting a zoom feature;

the processor determining a focal distance based on the requested zoom feature and the current focal region;

the processor selecting an adjustment value for the variable focus lens for a focal length about the focal distance; and

a focal region adjuster adjusting the variable focus lens in the see-through display based on the selected adjustment value.

3. The system of claim 2, wherein the variable focus lens is a liquid lens.

4. The system of claim 3, wherein:

the focal region adjuster focuses the variable focus lens by adjusting the radius of curvature of the liquid lens to adjust the focal length of the lens to be about the focal distance determined for the zoom feature.

5. In an augmented reality system, a method for providing improved focus of objects comprising:

determining a field of view of a see-through display of an augmented reality (AR) display device based on a three dimensional model of a space in which the AR display device is located, the space including a real object, and depth images of the space captured by one or more physical environment facing cameras supported by the AR display device;

determining a current focal region in the field of view; and

displaying a virtual object at its real world focal distance in the field of view by

identifying each virtual object which is in the current focal region based on its location in the three dimensional model,

placing each virtual object identified as being in the current focal region by a variable virtual focus adjuster for changing the focal region of a microdisplay assembly by changing an optical path between the microdisplay and one or more optical elements of the microdisplay assembly, the variable virtual focus adjuster rotating a rotable lens system,

generating an in-focus image including each identified virtual object in the current focal region by the microdisplay assembly,

optically receiving the in-focus image including each identified virtual object in the current focal region by the see-through display, and

displaying the in-focus image including each identified virtual object in the current focal region by the see-through display.

6. The method of claim 5, wherein:

determining the current focal region further comprises determining the current focal region based upon eye tracking data.

7. The method of claim 5, further comprising:

visually enhancing a real object in the field of view.

8. The method of claim 7, wherein:

visually enhancing the real object in the field of view further comprises;

identifying the real object in the field of view which satisfies enhancement criteria;

selecting an enhancement technique based on the enhancement criteria;

selecting real object image data used by the selected enhancement technique;

generating a virtual image implementing the selected enhancement technique; and

tracking the display of the virtual image to the real object based on the three dimensional model.

9. The method of claim 8, wherein:

the real object image data is edge data and the enhancement technique is one of the group consisting of;

edge enhancement of the at least one real object;

highlighting of the at least one real object; and

color enhancement of the at least one real object.

10. The method of claim 9, wherein:

the enhancement criteria is motion criteria for the real object.

11. The method of claim 5, further comprising:

displacing at least one lens of the rotatable lens system by

determining a first displacement value (S₁) between a front nodal point of the rotatable lens system and a target location of the virtual image,

determining a second displacement value (S₂) the microdisplay and a rear nodal point of the rotatable lens system for a focal length based on the equation;

1/S₁+1/S₂=1/f; and

applying at least one drive signal to the variable virtual focus adjuster to move at least one lens of the rotatable lens system to generate the determined first and second displacement values.

12. The method of claim 11, further comprising:

selecting the first displacement value based on the target location of the virtual image and the equation; and

applying the at least one drive signal to cause the variable virtual focus adjuster to change polarization of at least one birefringent lens in the microdisplay assembly to change the focal length to generate a selected value.

13. The method of claim 11, further comprising:

selecting the first displacement value based on the target location of the virtual image and the equation; and

applying the at least one drive signal to cause the variable virtual focus adjuster to effect a volume change in at least one lens, the at least one lens configured as a liquid lens, to change a radius of curvature to adjust the focal length to generate a selected value.