Comprehension and intent-based content for augmented reality displays

• US 9,213,405 B2
• Filed: 12/16/2010
• Issued: 12/15/2015
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Perez, Katie Stone, Bar-Zeev, Avi, Katz, Sagi
• Primary Examiner(s): HOLDER, ANNER N
• Application Number:
  US12/970,695
• Publication Number:
  US 20120154557A1
• Time in Patent Office: 1,825 Days
• Field of Search: 348/53, 345/419, 345/423
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94

Abstract

A method and system that enhances a user'"'"'s experience when using a near eye display device, such as a see-through display device or a head mounted display device is provided. The user'"'"'s intent to interact with one or more objects in the scene is determined. An optimized image is generated for the user based on the user'"'"'s intent. The optimized image is displayed to the user, via the see-through display device. The optimized image visually enhances the appearance of objects that the user intends to interact with in the scene and diminishes the appearance of objects that the user does not intend to interact with in the scene. The optimized image can visually enhance the appearance of the objects that increase the user'"'"'s comprehension. The optimized image is displayed to the user, via the see-through display device.

Claims

1. A method for generating an optimized image based on user intent in an augmented reality system, the method comprising:

determining a plurality of real world objects in a scene in a field of view of a see-through display device;

determining the field of view of a user through the see-through display device in the scene and determining a focal region of the user within the field of view;

determining the user'"'"'s intent to interact with one or more objects in the focal region of the user;

generating an optimized image, based on the user'"'"'s intent; and

displaying the optimized image to the user, the optimized image comprising a visual enhancement of at least one of the plurality of real world objects in the focal region of the user, and a simultaneous diminished appearance of one or more of the plurality of objects that are outside the focal region of the user but within the field of view of the user through the see-through display device.

2. The method of claim 1, wherein the step of determining one or more objects of interest includes creating a three dimensional model of the scene and further comprises:

segmenting the three dimensional model into one or more of the objects comprising the scene and identifying the one or more objects in the scene.

3. The method of claim 2, wherein the simultaneous diminished appearance comprises:

An increased opacity in the field of view containing said remaining plurality of real world objects.

4. The method of claim 1, wherein:

the focal region of the user is a Panum'"'"'s fusional area of eyes of the user.

5. The method of claim 1, wherein determining the user'"'"'s intent to interact with one or more objects in the focal region of the user further comprises:

detecting eye movement patterns of the user in the focal region of the user;

determining an intensity of gaze of the user on one or more objects being viewed by the user in the focal region of the user; and

inferring the user'"'"'s intent to interact with the one or more objects in the focal region of the user, based on the eye movement patterns and the intensity of the user'"'"'s gaze.

6. The method of claim 5, wherein determining the user'"'"'s intent to interact with one or more objects in the focal region of the user further comprises:

where said inferring is not determinative of the user'"'"'s intent to interact,

accessing user-specific information related to the user;

correlating one or more of the objects in the focal region of the user to the user-specific information; and

automatically inferring the user'"'"'s intent to interact with the one or more objects in the focal region of the user, based on the correlation.

7. The method of claim 6, wherein determining the user'"'"'s intent to interact with one or more objects in the focal region of the user further comprises:

where said automatically inferring is not determinative of the user'"'"'s intent to interact,

prompting the user to specify intent to interact with one or more of the objects in the focal region of the user;

receiving the user'"'"'s intent via at least one of a user physical action; and

determining the user'"'"'s intent to interact with the one or more objects based on the user physical action.

8. The method of claim 1, wherein generating the optimized image further comprises:

retrieving at least one of augmented content or audio information related to the one or more objects in the focal region of the user; and

projecting the augmented content or the audio information over or next to the one or more objects in the focal region of the user.

9. The method of claim 8, wherein visually enhancing the appearance of the one or more objects further comprises at least one of highlighting edges of the one or more objects or displaying a virtual box, a circle or a visual indicator in a region in which the one or more objects are located.

10. The method of claim 1 wherein the step of determining one or more objects of interest includes at least one of:

detecting an emitted signal from an object to the see-through display device; and

determining distanced of possible objects from the see-through display device using a distance detection technology.

11. The method of claim 8, further comprising displaying additional augmented content to a user, based on determining the user'"'"'s intent to interact with the augmented content displayed in the optimized image, wherein displaying the additional augmented content further comprises displaying a menu associated with the optimized image:

determining the user'"'"'s intent to interact with one or more of choices displayed by the menu;

receiving a selection of a choice in the menu, based on the user'"'"'s intent;

retrieving additional augmented content related to the choice; and

projecting the additional augmented content in the optimized image to the user.

12. An apparatus including a see-through display and a capture device, the apparatus including one or more processor readable storage devices having processor readable code embodied on said one or more processor readable storage devices, the processor readable code for programming one or more processors to perform a method comprising:

scanning a scene using the capture device to determine a plurality of real world objects of interest in the scene;

determining a field of view of the see-through display device in the scene, and the real world objects within the field of view;

determining a focal region of a user within the field of view;

identifying at least one of the plurality of real world objects in the focal region and a remaining plurality of said plurality of real world objects outside the focal region in an environment;

generating an optimized image in the field of view, the optimized image comprising a visual enhancement of the at least one of the plurality real world objects in the focal region of the user, and a simultaneous diminished appearance of any remaining plurality of real world objects that are outside the focal region of the user; and

displaying the optimized image in the field of view of the see-through display device.

13. One or more processor readable storage devices according to claim 12, wherein identifying the one or more objects in the focal region of the user that enhance user comprehension of the environment comprises gathering information about the environment, in real time.

14. One or more processor readable storage devices according to claim 12, wherein identifying the one or more objects in the focal region is based on user-specific information related to the user.

15. One or more processor readable storage devices according to claim 12, wherein generating the optimized image further comprises:

visually enhancing the appearance of the one or more objects that enhance comprehension in the environment and increasing opacity in the field of view containing said remaining pluralist of real world objects.

16. One or more processor readable storage devices according to claim 15, wherein visually enhancing the appearance of the one or more objects further comprises at least one of highlighting edges of the one or more objects or displaying a virtual box, a circle or a visual indicator in a region in which the one or more objects are located.

17. One or more processor readable storage devices according to claim 16, further comprising:

determining user intent to interact with the at least one of the plurality of real world objects in the optimized image, wherein determining user intent comprises;

determining an intensity of gaze of the user on the visual enhancement;

accessing user-specific information related to the user; and

determining the user'"'"'s intent to interact with the visual enhancement based on at least one of the intensity of gaze and the user-specific information.

18. An augmented reality system comprising:

a see-through display device having a field of view;

a micro display assembly attached to the see-through display device adapted to display an optimized image to a user, based on determining user intent to interact with one or more objects in a field of view of the device;

a processor communicatively coupled to the micro display assembly;

a memory accessible by the processor and storing software and data;

under control of software, the processor determines a plurality of real world objects in the field of view, the processor determines a user intent to interact with one of the plurality real world objects in a focal region of the user in the field of view by first determining an intensity of gaze of the user on the real world object through the see-through display device and if said intensity does not determine user intent, then accessing user-specific information related to the user, and determining the user'"'"'s intent to interact based a correlation between the real world objects in field of view and the user-specific information, and the processor generates an optimized image of the one of the plurality of real world objects in the field of view, the optimized image comprising a visual enhancement of the one of the plurality of real world objects in the focal region, and a simultaneous diminished appearance of any of the plurality of real world objects that are outside the focal region of the user but within the field of view.

19. The augmented reality system of claim 18, wherein the processor determines the user'"'"'s intent to interact with one or more objects in the focal region of the user based on at least one of eye movement patterns of the user, the intensity of the user'"'"'s gaze and user-specific information related to the user.

20. The augmented reality system of claim 18, wherein the micro display assembly highlights the one or more objects that the user intends to interact with, in the optimized image and increases opacity in the field of view containing said remaining plurality of real world objects.