Augmenting a field of view in connection with vision-tracking

• US 8,494,215 B2
• Filed: 03/05/2009
• Issued: 07/23/2013
• Est. Priority Date: 03/05/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kimchi, Gur, Bar-Zeev, Avi, Brewer, Brett D., Ormandy, Roman
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US12/398,280
• Publication Number:
  US 20100226535A1
• Time in Patent Office: 1,601 Days
• Field of Search: 382100-107, 345/7, 345/8, 345/156, 345/419, 345/420, 345/427, 345/632, 345/633, 345/629, 702/150, 348/135, 348/E7.085, 348/E13.023, 348/E13.041, 348/E13.002, 348/E13.014, 701/45
• US Class Current: 382/103
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2380/02 Flexible displays

Abstract

The claimed subject matter relates to an architecture that can employ vision-monitoring techniques to enhance an experience associated with elements of a local environment. In particular, the architecture can establish gaze- or eye-tracking attributes in connection with a user. In addition, a location and a head or face-based perspective of the user can also be obtained. By aggregating this information, the architecture can identify a current field of view of the user, and then map that field of view to a modeled view in connection with a geospatial model of the environment. In addition, the architecture can select additional content that relates to an entity in the view or a modeled entity in the modeled view, and further present the additional content to the user.

Claims

1. A computer implemented system that utilizes vision-tracking to enhance an experience associated with a field of view of a local environment, comprising:

an interface component that includes a vision-tracking component that monitors the location of one or more retinas of an associated user, and translates the location of one or more retinas into gaze attributes that relate to features of one or more eyes of an associated user;

a recognition component that employs the gaze attributes, an indication of spatial location, and an indication of directional perspective to identify a modeled view of a geospatial model of a physical environment, the modeled view corresponds to a current field of view of the user in connection with the physical environment; and

an augmentation component that selects additional content associated with a modeled entity included in the modeled view, the modeled entity is included in the geospatial model or an image-based data store; and

the interface component presents the additional content in a manner that is perceptible by the user.

2. The system of claim 1, the indication of location relates to a three-dimensional (3D) location of the user or an associated device.

3. The system of claim 1, the indication of perspective relates to a 3D orientation for the user or an associated device.

4. The system of claim 1, the vision-tracking component employs machine learning techniques to tailor the gaze attributes to a particular user'"'"'s eye or eyes.

5. The system of claim 1, the interface component is included in whole or in part in a head set or a wearable device or unit.

6. The system of claim 1, the interface component further includes a camera that images a view of the physical environment that encompasses the current field of view of the user.

7. The system of claim 1, the interface component includes or provides at least one of a head up display (HUD) or a visual retinal display (VRD).

8. The system of claim 6, the interface component presents the additional content by way of the HUD or the VRD.

9. The system of claim 1, the recognition component further determines an object of focus based upon the gaze attributes, the indication of location, and the indication of perspective, the object of focus in an entity included in the field of view.

10. The system of claim 9, the additional content is specific to the entity or the modeled entity associated therewith.

11. The system of claim 9, the entity is an environment entity and the recognition component identifies the modeled entity included in the geospatial model that corresponds to the environment entity.

12. The system of claim 9, the additional content is an audio stream associated with the entity or the modeled entity.

13. The system of claim 9, the interface component exposes a feature of the entity or the modeled entity, the feature is occluded or non-existent in the current field of view.

14. The system of claim 9, the entity is an object entity and the recognition component identifies the modeled entity included in the image-based data store that corresponds to the object entity.

15. The system of claim 14, the object entity included in the image is a representation of at least one of a logo, a brand, or a trademark;

a sign, a poster, a menu, a billboard, or an advertisement;

a work with artistic, entertainment, or commercial value;

a book, a compact disc (CD), a digital versatile disc (DVD), a product, or product packaging;

or a residence, a public site, a commercial site, or a landmark.

16. The system of claim 1, the indication of location or the indication of perspective is determined based upon at least one of Global Positioning System (GPS), signal triangulation techniques, signal multipath fingerprinting techniques, Radio Frequency Identification (RFID), compass information, altimeter information, accelerometer information, a gyroscope or level, or a comparison of the view of the local environment with the model of the environment.

17. A computer implemented method for employing vision-tracking for improving an experience in connection with a field of view of a local environment, comprising:

employing a vision-tracking component or device for tracking a location of one or more retinas of one or more eyes of a user;

employing a processor for translating the location of the one or more retinas into a set of gaze attributes;

employing the gaze attributes, an indication of location, and an indication of direction to identify a modeled view of a geospatial model of the environment, whereby the modeled view corresponds to a current field of view of the user;

selecting additional content associated with a modeled entity included in the modeled field of view, the modeled entity is included in the geospatial model or an image-based data store; and

presenting the additional content in a manner perceivable to the user.

18. The method of claim 17, further comprising at least one of the following acts:

defining the indication of location in 3D coordinates based upon a physical location of the user or an associated device;

defining the indication of direction in 3D based upon an physical orientation of the user'"'"'s head or face or a device associated with the user;

employing machine learning techniques for personalizing the gaze attributes to a particular user'"'"'s eye or eyes;

including the vision-tracking component in a headset or wearable device or unit;

including in the device or unit a camera for imaging a view of the physical environment that is substantially encompasses the current field of view of the user;

including in the device or unit a VRD or providing a HUD;

or

presenting the additional content by way of the VRD or the HUD.

19. The method of claim 18, further comprising at least one of the following acts:

determining an object of focus included in the field of view based upon the gaze attributes, the indication of location, and the indication of direction;

selecting the additional content further based upon a relationship with the object of focus;

selecting the additional content from amongst audio content associated with the object of focus or the modeled entity that corresponds therewith;

or

exposing a feature of the object of focus or the modeled entity associated therewith, the feature is occluded or non-existent in the current field of view.

20. A computer implemented system that utilizes vision-tracking to enhance an experience associated with a field of view of a local environment, comprising:

an interface component that includes a vision-tracking component that monitors one or more retinas of a user to determine gaze attributes;

a recognition component that employs the gaze attributes, an indication of spatial location, and an indication of directional perspective to identify a modeled view of a geospatial model of a physical environment, the modeled view corresponds to a current field of view of the user in connection with the physical environment;

an augmentation component that selects additional content associated with a modeled entity included in the modeled view, the modeled entity is included in the geospatial model; and

a virtual retina display that draws a raster display on the eye of the user, the raster display comprising the additional content and the modeled entity.