DISPLAYING A COLLISION BETWEEN REAL AND VIRTUAL OBJECTS

• US 20130286004A1
• Filed: 04/27/2012
• Published: 10/31/2013
• Est. Priority Date: 04/27/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: McCulloch, Daniel J., Latta, Stephen G., Mount, Brian J., Geisner, Kevin A., Scott, Jason, Steed, Jonathan T., Sugden, Ben J., Lee, Kyungsuk David, Finocchio, Mark J., Kipman, Alex Aben-Athar, Margolis, Jeffrey N., Sylvan, Roger Sebastian Kevin, Navratil, Arnulfo Zepeda, Hummel, Britta Silke
• Granted Patent: US 9,183,676 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/21 Collision detection, intersection

Abstract

Technology is described for displaying a collision between objects by an augmented reality display device system. A collision between a real object and a virtual object is identified based on three dimensional space position data of the objects. At least one effect on at least one physical property of the real object is determined based on physical properties of the real object, like a change in surface shape, and physical interaction characteristics of the collision. Simulation image data is generated and displayed simulating the effect on the real object by the augmented reality display. Virtual objects under control of different executing applications can also interact with one another in collisions.

Claims

1. A method for displaying a collision between a real object and a virtual object by an augmented reality display device system comprising:

identifying a collision between a real object and a virtual object in a display field of view of an, augmented reality display based on a respective three dimensional (3D) space position associated with each object in the display field of view;

determining at least one effect on at least one physical property of the real object due to the collision based on one or more physical properties of the real object and physical interaction characteristics for the collision;

generating image data of the real object simulating the at least one effect on the at least one physical property of the real object; and

displaying the image data of the real object registered to the real object.

2. The method of claim 1, the physical interaction characteristics including a velocity of at least one of the real object and the virtual object in the display field of view.

3. The method of claim 1 further comprising:

determining at least one effect on at least one physical property of the virtual object due to the collision based on its physical properties and the physical interaction characteristics for the collision;

modifying image data of the virtual object for representing the at least one effect on the at least one physical property of the virtual object; and

displaying the modified image data of the virtual object.

4. The method of claim 1 wherein determining the at least one effect on at least one physical property of the real object due to the collision based on one or more physical properties of the real object and physical interaction characteristics for the collision further comprises

simulating the collision by a physics engine based on a physics model for the real object, a physics model for the virtual object, the physical interaction characteristics for the collision and physics rules for a current physics environment.

5. The method of claim 1 further comprising:

receiving a 3D mapping of a user environment including a 3D space position and an identifier for the real object and a 3D space position and an identifier for the virtual object;

retrieving a stored physics model for the real object based on the identifier for the real object; and

retrieving a stored physics model for the virtual object based on the identifier for the virtual object.

6. The method of claim 1 further comprising:

identifying the real object in image data of the display field of view;

determining one or more physical properties of the real object based on the image data and object reference data sources; and

responsive to not finding an existing physics model for the real object, automatically generating a physics model for the real object based on the one or more physical properties.

7. The method of claim 6 wherein automatically generating a physics model for the real object based on the one or more physical properties further comprises

generating and storing a structural model based on a 3D shape physical property of the real object;

associating one or more collision effects with the real object in the physics model based on the type of object and one or more types of material of the real object;

associating one or more actions with the real object in the physics model based on a type of object for the real object; and

automatically storing the physics model in a network accessible data store for physics models.

8. The method of claim 6 further comprising:

responsive to user input indicating a change in the one or more physical properties of the real object, generating a version of the physics model of the real object with the change in the one or more physical properties of the real object; and

the physics engine using the version of the physics model for simulating a subsequent collision.

9. An augmented reality display device system for displaying a realistic collision between a virtual object and a real object comprising:

an augmented reality display device including an augmented reality display having a display field of view;

one or more processors communicatively coupled to the augmented reality display for controlling the display;

the one or more processors communicatively coupled to at least one camera for receiving image and depth data including the display field of view captured by the at least one camera;

the one or more processors identifying a collision between a virtual object and a real object in the field of view based on the received image and depth data and a three dimensional space position of the virtual object in the display field of view;

the one or more processors executing a physics engine for simulating the collision based on a physics model representing one or more physical properties of the real object, a physics model representing one or more physical properties of the virtual object and physical interaction characteristics of the collision;

the one or more processors generating image data simulating the collision including at least one effect on at least one physical property of the real object; and

the augmented reality display displaying the image data registered to the real object.

10. The augmented reality display device system of claim 9 wherein the augmented reality display is a see-through, augmented reality display.

11. The augmented reality display device system of claim 9 further comprising:

the one or more processors modifying image data of the virtual object for representing at least one effect on at least one physical property of the virtual object due to the collision.

12. The augmented reality display device system of claim 9 wherein the at least one effect on the at least one physical property of the real object comprises an effect on a surface of the real object.

13. The augmented reality display device system of claim 9 further comprising:

the one or more processors executing an object recognition engine for identifying one or more physical properties for the real object based on captured image data of the real object and object reference data storing one or more properties of real objects; and

the one or more processors having access to one or more network accessible data stores storing the object reference data.

14. The augmented reality display device system of claim 10, the one or more physical properties comprise a type of object, one or more types of surface material, one or more types of inner material, an object mass, an object size, an object shape, and object structure data.

15. The augmented reality display device system of claim 9 wherein the physics model for the real object and the physics model for the virtual object respectively comprise a structural model of the object, object physical properties data, one or more actions associated with the respective object, and one or more collision effects associated with the respective object.

16. The augmented reality display device system of claim 14 further comprising:

earphones attached to the support structure;

the one or more software controlled processors being communicatively coupled to the earphones for controlling output of audio data; and

wherein the physics model for the real object and the physics model for the virtual object further comprise one or more visual effects and one or more sound effects.

17. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for displaying a collision between virtual objects by a augmented reality display device system, the method comprising:

communicating identification data for one or more shared virtual objects to one or more applications which lack control over the one or more shared virtual objects;

providing access to a physics model for each of the one or more shared virtual objects for collision processing;

communicating action and 3D space position data for the one or more shared virtual objects to the one or more applications which lack control over the one or more shared virtual objects; and

displaying from a user perspective any of the one or more shared virtual objects having a 3D space position in a display field of view.

18. The one or more processor readable storage devices of claim 17 further comprising that one or more applications controlling the one or more shared virtual objects are executing on at least one other augmented reality display device system in the user environment.

19. The one or more processor readable storage devices of claim 18 further comprising the one or more applications which lack control over the one or more shared virtual objects designating the one or more shared virtual objects as real objects for control processing.

20. The one or more processor readable storage devices of claim 17 wherein communicating action and 3D space position data for the one or more shared virtual objects to the one or more applications which lack control over the one or more shared virtual objects further comprises:

updating a 3D mapping of a user environment to include space position data of the one or more shared virtual objects.