Context adaptive user interface for augmented reality display

• US 10,019,962 B2
• Filed: 08/17/2011
• Issued: 07/10/2018
• Est. Priority Date: 08/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, James C., Andrews, Anton O., Vaught, Benjamin I., Maitlen, Craig R., Novak, Christopher M., Small, Sheridan Martin
• Primary Examiner(s): Guo, Xilin
• Application Number:
  US13/212,172
• Publication Number:
  US 20130044128A1
• Time in Patent Office: 2,519 Days
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G09G 3/3413 Details of control of colour illumination sources
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A user interface includes a virtual object having an appearance in context with a real environment of a user using a see-through, near-eye augmented reality display device system. A virtual type of object and at least one real world object are selected based on compatibility criteria for forming a physical connection like attachment, supporting or integration of the virtual object with the at least one real object. Other appearance characteristics, e.g. color, size or shape, of the virtual object are selected for satisfying compatibility criteria with the selected at least one real object. Additionally, a virtual object type and appearance characteristics of the virtual object may be selected based on a social context of the user, a personal context of the user or both.

Claims

1. A system for providing a user interface with a virtual object appearing in context with a real environment of a user using a see through, augmented reality display device system comprising:

a computer system communicatively coupled via a network to the see-through near-eye, augmented reality display device system for receiving image data of a real environment of a user, the computer system comprising one or more processors and a memory accessible by the one or more processors;

the one or more processors being configured to identify one or more appearance characteristics including surface characteristics of one or more real world objects in the real environment;

the one or more processor being configured to identify one or more types of realistic physical connections available for providing a realistic visual connectivity relationship based on any of the appearance characteristics identified for the one or more real world objects;

a set of graphical design rules being stored in the memory, the set of graphical design rules defining compatibility criteria between appearance characteristics including as between respective surface characteristics;

the one or more processors being configured to select a type of object for the virtual object and to select an anchor real world objects set from the one or more real world objects based on compatibility criteria of the set of graphical design rules for forming a realistic physical connection using one of the one or more of the identified types of realistic physical connection available;

the one or more processors being configured to select at least one virtual connecting member for realistic visual interposition between a member of the selected anchor real world objects set and the virtual object so as to provide said realistic visual connectivity relationship using the selected at least one virtual connecting member as being visually interposed between a member of the selected anchor real world objects set and the virtual object; and

the one or more processors being configured to generate display data for an image of the virtual object having the realistic physical connection with a selected member of the anchor real world objects set with the selected at least one virtual connecting member being visually interposed there between, the generated display data being usable by the see-through, near-eye, augmented reality display device system.

2. The system of claim 1 further comprising the computer system for receiving data from one or more applications executing in the see-through, near-eye, augmented reality display system for identifying content being represented by the virtual object for display.

3. The system of claim 2 further comprising the one or more processors having access to applications executing on the see-through, near-eye, augmented reality display device system of the user for determining at least one of a social context for the user or a personal context for the user.

4. The system of claim 2 wherein the one or more processors have access to one or more stored images of real world objects;

and

the one or more processors for identifying one or more appearance characteristics of one or more real world objects in the real environment based on the stored images.

5. The system of claim 1 wherein the set of graphical design rules for defining compatibility criteria between appearance characteristics comprise definitions of at least one of the following:

complimentary colors;

a texture associated with a type of material;

a geometric orientation of a surface of a type of object; and

a type of physical connection available for the surface of the type of object.

6. The system of claim 1 wherein the set of graphical design rules link one or more animation effects with a type of appearance characteristic.

7. Processor-readable storage implemented as hardware storage having plural and respectively addressable storage locations for simultaneously storing respective addressable data items including instructions encoded therein for causing one or more processors to execute a method causing display of virtual imagery that appears to be realistically and in context integrated with a real world object of a real environment viewable by a user using a see-through, augmented reality display system, the executed method comprising:

first identifying a real world object present in the viewable real environment and to which predetermined virtual imagery can be visually coupled by way of a visually interposed virtual connecting member such that the predetermined virtual imagery appears to be realistically and in context connectively integrated with the identified real world object;

second identifying a virtual connecting member usable for realistic connective disposition visually between the predetermined virtual imagery and the identified real world object so as to provide said realistic and in context, visually connective integration of the virtual imagery with the identified real world object;

generating display data for displaying the virtual connecting member as being visually connectedly interposed between the predetermined virtual imagery and the identified real world object; and

using the generated display data to cause display by the see-through, augmented reality display system of the predetermined virtual imagery as being realistically and in context, visually connectively integrated with the identified real world object.

8. The processor-readable storage media of claim 7 wherein:

said first identifying includes identifying one or more real world appearance characteristics including those of respective surfaces of one or more real world objects present in the real environment;

said second identifying includes identifying one or more types of realistic physical connections that may be used during display to provide visually realistic virtual connectivities between the predetermined virtual imagery and at least one of the real world objects based on at least one of the identified real world surface appearance characteristics of the one or more real world objects;

the method further comprising;

based on the identified types of realistic physical connections, selecting a virtual object type from one or more virtual object type candidates capable of realistically being realistically portrayed as having at least one of the identified types of realistic physical connections; and

selecting from an anchor candidates group of the one or more real world objects, a set whose members can each serve as a realistic physical connection providing anchor to be visually connected by way of the identified virtual connecting member to the predetermined virtual imagery based on the selected virtual object type, the members of the selected set of the one or more real objects being capable of forming a realistic physical connection with at least one of the virtual objects of the selected virtual object type and vice versa.

9. The processor-readable storage media of claim 8 wherein the selecting of the virtual object type and the selecting of the anchor real world objects set comprises:

selecting the anchor real world objects set based on positions of the real world objects of the anchor real world object set with respect to a user field of view of the see-through, augmented reality display device system.

10. The processor-readable storage media of claim 8 wherein the method further comprises:

identifying a social context of the user based on user profile data; and

selecting the virtual object type based on the social context of the user.

11. The processor-readable storage media of claim 10 wherein the method further comprises:

selecting a user interface method of communicating output to a user based on the social context.

12. The processor readable storage media of claim 9 wherein the selecting of the anchor real world objects set based on a position of the anchor real world objects further comprises selecting the anchor real world objects set based on the positions of the objects of the anchor real world object set satisfying at least one of a social context position criteria or a personal context position criteria.

13. The processor-readable storage media of claim 10 wherein the method further comprises:

selecting at least one other appearance characteristic for the virtual object for satisfying appearance compatibility criteria with the anchor real world object set and based on the social context; and

generating display data for the image of the virtual object including the selected at least one other appearance characteristic.

14. The processor-readable storage of claim 7 wherein the second identifying identifies the virtual connecting member as being usable for providing realistic physical support for the predetermined virtual imagery to which the virtual connecting member visually connects.

15. A method for presenting predetermined virtual imagery that appears to be realistically and in context integrated with a real environment viewable by a user using a see-through, augmented reality display system, the method comprising:

first identifying as an anchor at least one real world object present in the real environment and viewable through the see-through, augmented reality display system where the identified anchor can serve as a foundation for realistically supporting a virtual connecting member to which the predetermined virtual imagery will be displayed as being physically connected to by way of interposition of the virtual connecting member such that the predetermined virtual imagery is seen as being realistically supported by both of the identified anchor and the interposed virtual connecting member;

second identifying one of a plurality of predetermined virtual connecting members as the virtual connecting member to be displayed and used for realistic visual and supportive connective disposition between the predetermined virtual imagery and the identified anchor so as to provide said realistic and in context integration of the virtual imagery with the identified anchor;

generating display data for displaying the identified virtual connecting member as being visually connectedly interposed between the predetermined virtual imagery and the identified real world object; and

using the generated display data to cause display by the see-through, augmented reality display system of the predetermined virtual imagery as being realistically and in context integrated with the identified real world object by way of the identified virtual connecting member.

16. The method of claim 15 wherein:

said first identifying includes identifying one or more real world appearance characteristics of one or more real world objects present in the real environment, the identified characteristics including those of respective surfaces of one or more real world objects present in the real environment;

said second identifying includes identifying one or more types of realistic physical connections that may be provided by a subset of the plurality of predetermined virtual connecting members so as to present realistic virtual connectivities between the predetermined virtual imagery and at least one of the real world objects based on at least one of the identified real world surface appearance characteristics of the one or more real world objects;

the method further comprising;

selecting from among the identified types a type that can serve as a realistic connecting member based on content of the predetermined virtual imagery;

selecting an anchor real world objects set from the one or more real world objects viewable in the real environment based on their capability of providing realistic physical support for the content of the predetermined virtual imagery.

17. The method of claim 16 further comprising:

using at least one other appearance characteristic for identifying the virtual connecting member for satisfying appearance compatibility criteria with the anchor real world object set.

18. The method of claim 17 wherein the one or more appearance characteristics used for identifying the virtual connecting member comprises at least one of the following:

a size, a shape, a color, a surface, a geometric orientation of a surface, a surface texture, a pattern, and physical interaction characteristics.

19. The method of claim 18 wherein the physical connection type available for the surface comprises at least one of the following:

attachment;

hanging;

integration; and

supporting.

20. The method of claim 16 further comprising:

replacing a real world object with a virtual object in a user field of view of the see-through, augmented reality display device system.

21. The method of claim 16 further comprising

identifying a personal context of the user including a state of being based on eye data of the user;

and

wherein selecting the type of object for the virtual connecting member further comprises selecting the virtual connecting member based on the personal context of the user.

22. The method of claim 21 further comprising

selecting at least one other appearance characteristic of the virtual connecting member in the image for display based on a social context of a user determined from image data of a physical location of the user.

23. The method of claim 22 wherein identifying a social context of the user based on user profile data further comprises determining the social context based on data identifying at least one of the following:

a person in a vicinity of the user;

a geographic location of the user; and

an activity of the user.