CONTEXT ADAPTIVE USER INTERFACE FOR AUGMENTED REALITY DISPLAY

• US 20130044128A1
• Filed: 08/17/2011
• Published: 02/21/2013
• Est. Priority Date: 08/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, James C., Andrews, Anton O., Vaught, Benjamin I., Maitlen, Craig R., Small, Sheridan Martin, Novak, Christopher M.
• Granted Patent: US 10,019,962 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G09G 3/3413 Details of control of colour illumination sources
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A user interface includes a virtual object having an appearance in context with a real environment of a user using a see-through, near-eye augmented reality display device system. A virtual type of object and at least one real world object are selected based on compatibility criteria for forming a physical connection like attachment, supporting or integration of the virtual object with the at least one real object. Other appearance characteristics, e.g. color, size or shape, of the virtual object are selected for satisfying compatibility criteria with the selected at least one real object. Additionally, a virtual object type and appearance characteristics of the virtual object may be selected based on a social context of the user, a personal context of the user or both.

Claims

1. One or more processor-readable storage media having

instructions encoded thereon for causing one or more processors to execute a method for providing a user interface with a virtual object appearing in context with a real environment of a user using a see-through, augmented reality display system, the method comprising:

identifying one or more appearance characteristics including a surface of one or more real world objects in the real environment;

identifying one or more types of physical connections available based on any surface identified for the one or more real world objects;

selecting a virtual object type from one or more virtual object candidates and an anchor real world object set from the one or more real world objects based on the virtual object type and the one or more real objects of the anchor set being capable of forming a physical connection with each other; and

generating display data for an image of the virtual object having the selected virtual object type and forming the physical connection with the anchor real world object set.

2. The one or more processor-readable storage media of claim 1

wherein selecting the virtual object type and the anchor real world object set from the one or more real world objects based on the virtual object type and the one or more real objects of the anchor set being capable of forming the physical connection with each other further comprises:

selecting the anchor real world object set based on a position of the anchor real world object set with respect to a user field of view of the see-through, augmented reality display device system.

3. The one or more processor-readable storage media of claim 1

wherein the method further comprises:

identifying a social context of the user based on user profile data; and

selecting the virtual object type based on the social context of the user.

4. The one or more processor-readable storage media of claim 3

wherein the method further comprises:

selecting a user interface method of communicating output to a user based on the social context.

5. The one or more processor readable storage media of claim 2

wherein selecting the anchor real world object set based on a position of the anchor real world object with respect to a user field of view of the see-through, augmented reality display device system further comprises selecting the anchor real world object set based on the position of the anchor real world object set satisfying at least one of a social context position criteria or a personal context position criteria.

6. The one or more processor-readable storage media of claim 3

wherein the method further comprises:

selecting at least one other appearance characteristic for the virtual object for satisfying appearance compatibility criteria with the anchor real world object set and based on the social context; and

generating display data for the image of the virtual object including the selected at least one other appearance characteristic.

7. A method for providing a user interface with a virtual object for

displaying content, the virtual object appearing in context with a real environment of a user using a see through, augmented reality display device system comprising:

identifying one or more appearance characteristics including a surface of one or more real world objects in the real environment;

identifying one or more types of physical connections available based on any surface identified for the one or more real world objects;

selecting a type of object for the virtual object based on content to be represented by the virtual object and the type of object being capable of forming a physical connection using at least one of the one or more types of physical connections available;

selecting an anchor real world object set from the one or more real world objects in the real environment based on capability of forming the physical connection with the selected virtual object; and

generating display data for an image of the virtual object having the virtual object type selection and forming the physical connection with the anchor real world object set.

8. The method of claim 7 further comprising:

selecting at least one other appearance characteristic for the virtual object for satisfying appearance compatibility criteria with the anchor real world object set; and

generating display data for the image of the virtual object including the selected at least one other appearance characteristic satisfying the appearance compatibility criteria with the anchor real world object set.

9. The method of claim 8 wherein the one or more appearance

characteristics of the virtual object or of the real object comprises at least one of the following:

a size, a shape, a color, a surface, a geometric orientation of a surface, a surface texture, a pattern, and physical interaction characteristics.

10. The method of claim 9 wherein the physical connection type

available for the surface comprises at least one of the following:

attachment;

hanging;

integration; and

supporting.

11. The method of claim 7 further comprising:

replacing a real world object with a virtual object in a user field of view of the see-through, augmented reality display device system.

12. The method of claim 7 further comprising

identifying a personal context of the user including a state of being based on eye data of the user;

and

wherein selecting the type of object for the virtual object further comprises selecting the virtual object based on the personal context of the user.

13. The method of claim 12 further comprising

selecting at least one other appearance characteristic of the virtual object in the image for display based on a social context of a user determined from image data of a physical location of the user.

14. The method of claim 13 wherein identifying a social context of

the user based on user profile data further comprises determining the social context based on data identifying at least one of the following:

a person in a vicinity of the user;

a geographic location of the user; and

an activity of the user.

15. A system for providing a user interface with a virtual object

appearing in context with a real environment of a user using a see through, augmented reality display device system comprising:

a computer system communicatively coupled via a network to the see-through near-eye, augmented reality display device system for receiving image data of a real environment of a user, the computer system comprising one or more processors and a memory accessible by the one or more processors;

the one or more processors for identifying one or more appearance characteristics including a surface of one or more real world objects in the real environment;

the one or more processor for identifying one or more types of physical connections available based on any surface identified for the one or more real world objects;

a set of graphical design rules being stored in the memory, the set of graphical design rules defining compatibility criteria between appearance characteristics;

the one or more processors for selecting a type of object for the virtual object and selecting an anchor real world object set from the one or more real world objects based on compatibility criteria of the set of graphical design rules for forming a physical connection using one of the one or more types of physical connection available; and

the one or more processors for generating display data for an image of the virtual object having the physical connection with the anchor real world object set by the see-through, near-eye, augmented reality display device system.

16. The system of claim 15 further comprising the computer system

for receiving data from one or more applications executing in the see-through, near-eye, augmented reality display system for identifying content being represented by the virtual object for display.

17. The system of claim 16 further comprising the one or more

processors having access to applications executing on the see-through, near-eye, augmented reality display device system of the user for determining at least one of a social context for the user or a personal context for the user.

18. The system of claim 16 wherein the one or more processors have

access to one or more stored images of real world objects;

and

the one or more processors for identifying one or more appearance characteristics of one or more real world objects in the real environment based on the stored images.

19. The system of claim 15 wherein the set of graphical design rules

for defining compatibility criteria between appearance characteristics comprise definitions of at least one of the following:

complimentary colors;

a texture associated with a type of material;

a geometric orientation of a surface of a type of object; and

a type of physical connection available for the surface of the type of object.

20. The system of claim 15 wherein the set of graphical design rules

link one or more animation effects with a type of appearance characteristic.