Object-centric mixed reality space

• US 9,767,720 B2
• Filed: 06/25/2012
• Issued: 09/19/2017
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kinnebrew, Peter Tobias, Kamuda, Nicholas Ferianc
• Primary Examiner(s): NGUYEN, ANH TUAN V
• Application Number:
  US13/532,636
• Publication Number:
  US 20130342570A1
• Time in Patent Office: 1,912 Days
• Field of Search: None
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

A see-through, near-eye, mixed reality display apparatus providing a mixed reality environment wherein one or more virtual objects and one or more real objects exist within the view of the device. Each of the real and virtual have a commonly defined set of attributes understood by the mixed reality system allowing the system to manage relationships and interaction between virtual objects and other virtual objects, and virtual and real objects.

Claims

1. A method presenting a mixed reality environment allowing viewing of real world objects integrated with virtual objects by a user, comprising:

determining one or more real objects viewable by the user within a user environment;

determining one or more three-dimensional virtual objects adapted to be rendered to the user and viewable in conjunction with the one or more real objects within the user environment;

rendering the one or more virtual objects within a user field of view within the environment;

mapping a real object of the one or more real objects and a three-dimensional virtual object of the one or more three-dimensional virtual objects to respective object instances, the three-dimensional virtual object and the real object in the object instances defined using the same object definition including the same object definition of a physical attribute of the one or more real objects and the one or more three-dimensional virtual objects; and

managing interaction between the three-dimensional virtual object and the real object based on the physical attribute defined in the respective object instances upon interaction of the three-dimensional virtual object and the real object.

2. The method of claim 1 wherein said rendering of the one or more virtual objects is based on at least one relational attribute in the same object definition defining a behavior of the one or more virtual objects relative to the one or more real objects.

3. The method of claim 1 wherein the step of managing interaction comprises:

tracking the real object and the virtual object within the user environment;

determining a virtual-real object interaction when the virtual object interacts with the real object; and

rendering the virtual-real object interaction based on the set of attributes for a virtual object instance and the attributes of a real object instance.

4. The method of claim 1 further including:

tracking each of the one or more real objects and each of the one or more virtual objects within the user environment;

determining a virtual-virtual object interaction when a virtual object interacts with another virtual object;

rendering the virtual-virtual object interaction based on a user filter;

determining a virtual-real object interaction when a virtual object interacts with a real object; and

rendering the virtual-real object interaction based on the set of attributes for a virtual object instance and the attributes of a real object instance.

5. The method of claim 1 wherein the set of attributes includes one or more functions for the virtual object.

6. The method of claim 1 further including generating the object instances and sharing object instances with other users via a communication link, and receiving shared object instances from other users, and including rendering and managing the interaction between the object instances shared by other users and generated object instances.

7. A see through head mounted display apparatus, comprising:

a see-through, near-eye, augmented reality display adapted to render three-dimensional virtual objects to the user in the display which are viewable in conjunction with real objects;

one or more processing devices in wireless communication with apparatus, the one or more processing devices automatically determine an environment, one or more real objects in the environment and one or more three-dimensional virtual objects in the environment, the one or more processing devices assign an object instance to each of the real and three-dimensional virtual objects in the environment, each three-dimensional virtual object and each real object in each object instance defined having an object definition representing a physical trait of the one or more three-dimensional virtual objects and the one or more real objects and provided in a data structure containing a common set of attributes for the real and three-dimensional virtual objects such that each three-dimensional virtual object and each real object share the same attributes, the one or more processing devices determine input data from real world objects and three-dimensional virtual objects in a field of view and integrate interaction between real and three-dimensional virtual objects based on the object instances;

wherein said interaction of the one or more three-dimensional virtual objects with one or more real objects is based on at least one relational attribute in the common set of attributes defining the behavior of the three-dimensional virtual object relative to the one or more real objects.

8. The apparatus of claim 7 wherein the common set of attributes comprises a data structure including includes at least one attribute of:

object type, spatial coordinates, object registration, reality rating, dynamic scaling, ownership, user permissions, content rating, physical properties, learned attributes, related objects and functions.

9. The apparatus of claim 8 wherein the physical properties include at least physics attributes defining object movement and actions and an interaction rule set defining object interaction with other objects.

10. The apparatus of claim 8 wherein the object definition includes an identifier.

11. The apparatus of claim 10 wherein each instance of an object definition is specifically identified.

12. The apparatus of claim 8 wherein the apparatus includes a memory and a data structure, the data structure including one or more object definitions modified by a user and owned by a user.

13. The apparatus of claim 12 wherein the one or more processors

track each of the real objects and each of the virtual objects within a user environment;

determine a virtual-virtual object interaction when a virtual object interacts with another virtual object;

render the virtual-virtual object interaction based on a user filter;

determine a virtual-real object interaction when a virtual object interacts with another virtual object; and

render the virtual-real object interaction based on the set of attributes for a virtual object instance and the attributes of a real object instance.

14. A method for managing interaction between virtual holographic objects and real world objects in a mixed reality environment generated by a see through head mounted display system, comprising:

determining an environment and orientation of the system, the system includes one or more sensors and a see-through display;

determining three-dimensional locations in three-dimensional space of real and virtual objects within an environment of a wearer of the see-through display in the environment;

creating an object instance for virtual objects and real objects within the environment, the virtual objects and the real objects in the object instances defined based on a common object definition comprising a set of attributes such that the virtual objects and the real objects share the same sub-set of core attributes, wherein the set of attributes of the common object definition relates to a common physical trait of three-dimensional virtual object and each real object;

determining whether an interaction between at least two objects occurs, the interaction being one of an interaction between a virtual object and another virtual object, or an interaction between a virtual object and a real world object, and the interaction comprising the at least two objects sharing at least one point in three-dimensional space;

rendering virtual objects to the display which are viewable in conjunction with the real objects where the interaction between the at least two objects in the display based on attributes defined in the object instance of any interacting virtual and real objects and a system filter, the system filter interpreting the attributes of each of the interacting objects according to user specified filter settings relative to rendering the interaction in the display and wherein said rendering of the one or more virtual objects is based on at least one relational attribute defining the behavior of the virtual object relative to the one or more real objects.

15. The method of claim 14 wherein each object instance contains a value for at least one attribute of:

object type, spatial coordinates, object registration, reality rating, dynamic scaling, ownership, user permissions, content rating, physical properties, learned attributes, related objects and functions, and the interaction between the at least two objects is based on an the value in each said attribute.

16. The method of claim 15 wherein the at least one relational attribute defines a relation of the object to a user, an environment or another object.

17. The method of claim 16 wherein the attribute of spatial coordinates defines the position of an object relative to a relational attribute value.

18. The method of claim 15 further including accessing generic object libraries provided by a mixed reality service, the generic object libraries containing generic object definitions accessible by an object identifier, the generic object definitions used to create object instances.

19. The method of claim 18 further including providing user-specific object definitions accessible by an object identifier to a mixed reality service, the user-specific object definitions used to create object instances by a creating user and a shared user.