Enhancing an object of interest in a see-through, mixed reality display device

• US 9,323,325 B2
• Filed: 08/30/2011
• Issued: 04/26/2016
• Est. Priority Date: 08/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Perez, Kathryn Stone, Vaught, Benjamin I., Lewis, John R., Crocco, Robert L., Kipman, Alex Aben-Athar
• Primary Examiner(s): Pe, Geepy
• Assistant Examiner(s): Joisil, Berteau
• Application Number:
  US13/221,770
• Publication Number:
  US 20130050432A1
• Time in Patent Office: 1,701 Days
• Field of Search: 345/661, 345/6, 345/419, 345/633, 345/8, 345/156, 348/47, 351/206
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

Technology is disclosed for enhancing the experience of a user wearing a see-through, near eye mixed reality display device. Based on an arrangement of gaze detection elements on each display optical system for each eye of the display device, a respective gaze vector is determined and a current user focal region is determined based on the gaze vectors. Virtual objects are displayed at their respective focal regions in a user field of view for a natural sight view. Additionally, one or more objects of interest to a user may be identified. The identification may be based on a user intent to interact with the object. For example, the intent may be determined based on a gaze duration. Augmented content may be projected over or next to an object, real or virtual. Additionally, a real or virtual object intended for interaction may be zoomed in or out.

Claims

1. A near-eye, mixed reality display system comprising:

at least one display optical system being positioned in a near-eye, mixed reality display device to be seen through by at least one eye;

at least one image generation unit for generating an image and having an optical alignment with the at least one display optical system, the at least one image generation unit having a variable focal length;

a gaze detection system comprising a respective arrangement of gaze detection elements positioned by the near-eye, mixed reality display device, the gaze detection elements including illuminators for generating glints and at least one sensor having a detection area for detecting glints for the at least one display optical system and generating eye data;

one or more software controlled processors communicatively coupled to the at least one image generation unit and the at least one sensor;

a memory accessible by the one or more processors for storing software and data including the eye data;

the one or more processors for determining at least one gaze vector for the at least one eye based on the eye data; and

the one or more software controlled processors operable to determine a current user focal region based on the at least one gaze vector in a current 3D user field of view, the one or more processors operable to identify one or more virtual objects in the current 3D user field of view that the current user intends to interact with in the current user focal region, and the one or more processors operable to control the focal length of the at least one image generation unit of the display optical system of the current user to create one or more images in which each of the one or more virtual objects appears at a respective focal region in the current 3D user field of view for a natural sight view, wherein

the object in the current user focal region is correlated with user-specific information from a user profile database such that the user intent to interact with the object is based on the correlation.

2. The system of claim 1 further comprising:

wherein the at least one sensor is an infra-red (IR) image sensor for providing infra-red image data of a pupil and the glints;

a first illuminator and the detection area of the IR image sensor lie in a first plane with a center of a cornea of the at least one eye;

a second illuminator and the detection area of the IR image sensor lie in a second plane with the center of the cornea of the at least one eye;

the arrangement of the illuminators and the detection area cause the first and second planes to intersect; and

the one or more software controlled processors determining a position of the center of the cornea from the intersection of the planes, identifying a center of the pupil from the image data, and wherein the one or more processors determining at least one gaze vector for the at least one eye based on the eye data further comprises determining a gaze vector for the at least one eye based on the position of the center of the cornea and the center of the pupil.

3. The system of claim 1 further comprising:

wherein the at least one sensor comprises a position sensitive detector and the one or more processors turn each illuminator on and off in a predetermined sequence for capturing glint data.

4. The system of claim 1 further comprising:

each illuminator having a fixed position in the arrangement of gaze detection elements and each illuminator transmits a respective infra-red (IR) signal about a predetermined IR wavelength;

wherein the at least one sensor comprises an infra-red (IR) sensitive sensor for detecting intensity values for at least two glints at predetermined positions on the IR sensitive sensor;

the one or more processors operable to determine a pupil position for the at least one eye based on determining a reflectivity data value for each glint intensity value, and identifying the glint as being located on a sclera, an iris portion or a pupil portion based on the reflectivity data value for each glint intensity value; and

wherein the one or more processors for determining at least one gaze vector for the at least one eye based on the eye data further comprises the one or more processors operable to determine the gaze vector based on the pupil position.

5. The system of claim 1 further comprising:

the near-eye, mixed reality display device being a head mounted display device including an image camera for capturing image data of one or more real objects in front of the user;

one or more depth cameras for capturing three dimensional image data for positions of one or more real objects in a physical environment surrounding the user;

the memory storing the image data captured by the one or more image and depth cameras; and

the one or more software controlled processors being operable for generating a three-dimensional model of an augmented reality scene based on locations of the one or more real objects derived from the image data captured by the cameras and locations of one or more virtual objects based on an executing software application.

6. The system of claim 5 further comprising:

the near-eye, mixed reality display device being a pair of eyeglasses and including an orientation sensor on a temple of the eyeglasses; and

the one or more processors being operable for determining a head position based on orientation data from the orientation sensor and the current user field of view based on the head position.

7. The system of claim 6 further comprising:

wherein the at least one image generation unit is a microdisplay assembly including a variable focus adjuster; and

wherein the one or more software controlled processors operable to control the focal length of the at least one image generation unit to create one or more images in which each of the one or more virtual objects appears at a respective focal region in the current 3D user field of view for the natural sight view further comprises;

the one or more processors operable to control the variable focus adjuster of the microdisplay assembly to change the focal length in an optical alignment between the microdisplay assembly and the at least one display optical system.

20. The system of claim 1, wherein the at least one sensor is a discrete surface position sensitive detector which is connected set of pixels have intensity values within a predetermined range and are pruned by identifying as a candidate glint each connected set of pixels which satisfy glint geometry criteria.

8. A method for generating an optimized image based on an object of interest in a system including a near-eye mixed reality display device, the method comprising:

determining a three dimensional (3D) current user field of view of a user wearing the near-eye, mixed reality display device, the current user field of view including one or more real objects;

determining at least one gaze vector for at least one eye based on an arrangement of gaze detection elements in fixed positions with respect to each other positioned by the near-eye, mixed reality display device for at least one display optical system positioned to be seen through by the at least one eye by the display device;

determining a current user focal region based on the at least one gaze vector within the 3D current user field of view;

displaying one or more images including one or more virtual objects to appear at a respective focal region in the current 3D user field of view for a natural sight view;

identifying an object of interest by determining a user intent to interact with the object in the current user focal region;

generating an optimized image based on the object of interest; and

displaying the optimized image within the current user focal region to the user and displaying images appearing outside of the current user focal region and within the current 3D user field of view that the user does not intend to interact with as diminished using an opacity filter, via the near-eye mixed reality display device.

9. The method of claim 8 wherein the arrangement of gaze detection elements comprises:

an image sensor having a detection area facing the at least one eye, and the at least one gaze vector is determined based on image data captured by the image sensor.

10. The method of claim 8 wherein determining the user intent to interact with the object further comprises:

detecting eye gaze patterns of the user in the current user focal region;

determining a gaze duration of the user for the object in the current user focal region based on a time period a point of gaze is maintained in the eye gaze patterns on the object, the point of gaze being determined based on the at least one gaze vector; and

determining the user intent to interact with the object based on whether the gaze duration of the user for the object satisfies a gaze duration threshold.

11. The method of claim 8, wherein determining the user intent to interact with the object in the current user focal region further comprises:

accessing user-specific information related to the user;

correlating the object in the current user focal region to the user-specific information; and

automatically determining the user intent to interact with the object in the current user focal region, based on the correlation.

12. The method of claim 8, wherein generating the optimized image based on the object of interest further comprises:

retrieving augmented content related to the object; and

projecting the augmented content over or next to the object in the current user focal region.

13. The method of claim 8, wherein generating the optimized image based on the object of interest further comprises:

visually enhancing an appearance of the object in the current user focal region, that the user intends to interact with; and

diminishing an appearance of one or more objects that are outside the current user focal region but within the current user field of view.

14. The method of claim 13, wherein visually enhancing the appearance of the object further comprises at least one of highlighting edges of the object or displaying a visual indicator in a region in which the object is located.

15. A method for enhancing a display view of an object of interest in a near-eye, mixed reality display device, the method comprising:

determining a current three dimensional (3D) user field of view of a user wearing the near-eye, mixed reality device, the current 3D user field of view including one or more real objects;

determining a gaze vector for each user eye based on an arrangement of gaze detection elements in fixed positions with respect to each other on a respective display optical system for each eye of the display device;

determining a current user focal region based on the at least one gaze vector within the current user field of view;

displaying one or more images including one or more virtual objects appearing at a respective focal region in the current 3D user field of view for a natural sight view;

identifying an object of interest to the user in the current 3D user field of view; and

visually enhancing an appearance of the object of interest by adjusting a focal region of the object in the current 3D user field of view of a user wearing the near-eye, mixed reality display device, wherein adjusting the focal region includes changing the focal point and the focal region of the object of interest to a zoom focal region in response to receipt of a zoom request.

16. The method of claim 15 wherein:

the object of interest is a virtual object.

17. The method of claim 15 wherein:

the object of interest is a real object; and

visually enhancing the appearance of the object of interest by adjusting the focal region of the object of interest in the current 3D user field of view of a user wearing the near-eye, mixed reality display device further comprises;

receiving a zoom request from the user;

determining a focal distance for a zoom focal region based on the zoom request and the current user focal region; and

adjusting a respective variable focus lens in each respective display optical system for focusing the display device in the zoom focal region.

18. The method of claim 17 further comprising:

capturing image data of a physical environment of the user by a front facing camera on the near-eye display device at a zoom setting with a zoom lens; and

tracking the zoom setting of the front facing camera to the zoom focal region.

19. The method of claim 18 further comprising:

updating the current user field of view based on the zoom focal region and image data from one or more cameras in a physical environment of a user capturing image data of a physical environment of the user including the user.