Displaying a collision between real and virtual objects

• US 9,183,676 B2
• Filed: 04/27/2012
• Issued: 11/10/2015
• Est. Priority Date: 04/27/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: McCulloch, Daniel J., Latta, Stephen G., Mount, Brian J., Geisner, Kevin A., Sylvan, Roger Sebastian Kevin, Navratil, Arnulfo Zepeda, Scott, Jason, Steed, Jonathan T., Sugden, Ben J., Hummel, Britta Silke, Lee, Kyungsuk David, Finocchio, Mark J., Kipman, Alex Aben-Athar, Margolis, Jeffrey N.
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US13/458,800
• Publication Number:
  US 20130286004A1
• Time in Patent Office: 1,292 Days
• Field of Search: 345/419, 345/420
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/21 Collision detection, intersection

Abstract

Technology is described for displaying a collision between objects by an augmented reality display device system. A collision between a real object and a virtual object is identified based on three dimensional space position data of the objects. At least one effect on at least one physical property of the real object is determined based on physical properties of the real object, like a change in surface shape, and physical interaction characteristics of the collision. Simulation image data is generated and displayed simulating the effect on the real object by the augmented reality display. Virtual objects under control of different executing applications can also interact with one another in collisions.

Claims

1. A method for displaying a simulated collision between a real object and a virtual object using a see-through augmented reality display device comprising:

for real and virtual objects present in a displayed field of view of the see-through augmented reality display device, identifying at least one collision between a first real object having one or more parts that are viewable through a see-through portion of the see-through augmented reality display device and a first virtual object based on respective three dimensional (3D) augmented reality space positions associated with the first real and virtual objects;

determining at least one virtual-world deformative effect to be virtually imparted to a part of the first real object due to the virtual space collision based on one or more of determined pre-collision physical properties assigned to respective parts of the first real object and of determined physical interaction characteristics for the collision;

generating image data for the first real object simulating the at least one virtual-world deformative effect as having been virtually imparted to a collision affected part of the first real object, where the collision affected part of the first real object is not actually deformed by the collision with the first virtual object; and

displaying the generated image data including the at least one deformative effect virtually imparted to the collision affected part of the first real object using the see-through augmented reality display device, where the generated and displayed image data of the collision affected part is registered to a location of the first real object as seen using the see-through augmented reality display device.

2. The method of claim 1, wherein the physical interaction characteristics for the collision include a velocity of at least one of the first real object and of the first virtual object in the display field of view.

3. The method of claim 1 wherein the see-through augmented reality display device is a three-dimensional scene displaying head mounted display device, the method further comprising:

determining at least one change to at least one physical property of the first virtual object due to the collision based on its pre-collision physical properties and the determined physical interaction characteristics for the collision;

modifying image data of the first virtual object for representing the at least one change to its at least one physical property; and

displaying the modified image data of the virtual object in addition to said displaying of the generated image data including the at least one deformative effect virtually imparted to the first real object.

4. The method of claim 1 wherein the step of determining the at least one virtual-world deformative effect on the simulated version of the first real object due to the collision further comprises:

simulating the collision by use of a physics engine and based on a pre-collision physics model for the first real object, a pre-collision physics model for the first virtual object, on the determined physical interaction characteristics for the collision and on determined physics rules for a current physical environment of the first real and virtual objects.

5. The method of claim 1 further comprising:

receiving a 3D mapping of a user environment including a 3D space position and an identifier for the first real object and a 3D space position and an identifier for the first virtual object;

retrieving a stored, current physics model for the first real object based on the identifier for the first real object; and

retrieving a stored, current physics model for the first virtual object based on the identifier for the first virtual object.

6. The method of claim 1 further comprising:

identifying the first real object in image data of the display field of view;

determining one or more current physical properties of the first real object based on the image data and object reference data sources; and

responsive to not finding an existing physics model for the first real object, automatically generating a physics model for the first real object based on the one or more physical properties.

7. The method of claim 6 wherein automatically generating a physics model for the real object based on the one or more physical properties further comprises:

generating and storing a structural model based on a 3D shape physical property of the first real object;

associating one or more collision effects with the collision affected part of the first real object in the physics model based on the type of object and one or more types of material assigned to the collision affected part of the first real object;

associating one or more actions with the collision affected part of the first real object in the physics model based on a type of object for the first real object; and

automatically storing the physics model in a network accessible data store for physics models.

8. The method of claim 6 further comprising:

responsive to user input indicating a change in the one or more physical properties of the first real object, generating a version of the physics model of the first real object with the change in the one or more physical properties of the first real object; and

the physics engine using the version of the physics model for simulating a subsequent collision and a corresponding deformative effect virtually being imparted to the first real object.

9. The method of claim 1 wherein the at least one deformative effect virtually imparted to the first real object depicts the first real object as having been cracked.

10. The method of claim 1 wherein the at least one deformative effect virtually imparted to the first real object depicts the first real object as having been broken.

11. The method of claim 1 wherein the at least one deformative effect virtually imparted to the first real object depicts the first real object as having been bent.

12. An augmented reality display device system for displaying a realistic collision between a virtual object and a real object comprising:

an augmented reality display device including a see-through augmented reality display having a display field of view in which one or more parts of the real object are viewable as light passing through a see-through portion of the see-through augmented reality display;

one or more processors communicatively coupled to the augmented reality display for controlling the display;

the one or more processors being communicatively coupled to at least one camera for receiving image and depth data including the display field of view captured by the at least one camera;

the one or more processors being configured to identify a collision between a virtual object and the real object in the field of view based on the received image and depth data and a three dimensional space position of the virtual object in the display field of view;

the one or more processors being configured to implement a physics engine that simulates a virtualized collision in a virtual space based on a physics model representing one or more physical properties of a collision affected part of the real object, a physics model representing one or more physical properties of the virtual object and physical interaction characteristics of the collision;

the one or more processors being configured to generate image data simulating the collision including at least one virtual-world deformative effect imparted onto the collision affected part of the real object as a result of the virtualized collision; and

the augmented reality display being configured to display the generated image data for the affected part of the real object as being registered to a corresponding portion of the real object.

13. The augmented reality display device system of claim 12 further comprising:

the one or more processors being configured for modifying image data of the virtual object for representing at least one effect imparted on at least one physical property of the virtual object due to the virtualized collision with the real object.

14. The augmented reality display device system of claim 12 wherein the at least one virtual-world deformative effect imparted onto the registered image of the real object is selected from the group consisting of a cracking of the image-represented real object, a bending of the image-represented real object and a breaking apart of the image-represented real object.

15. The augmented reality display device system of claim 12 further comprising:

the one or more processors configured for implementing an object recognition engine for identifying one or more physical properties for the real object based on captured image data of the real object and object reference data storing one or more properties of real objects; and

the one or more processors having access to one or more network accessible data stores storing the object reference data.

16. The augmented reality display device system of claim 12 wherein the physics model for the real object and the physics model for the virtual object respectively comprise a structural model of the object, object physical properties data, one or more actions associated with the respective object, and one or more collision effects associated with the respective object.

17. One or more processor readable storage devices comprising

instructions which cause one or more processors to execute a method for displaying a collision between a real object and a virtual object by an augmented reality display device system, the augmented reality display device system including a see-through augmented reality display having a display field of view in which one or more parts of the real object are viewable as light passing through a see-through portion of the see-through augmented reality display, the method comprising:

communicating identification data for one or more shared virtual objects to one or more applications which lack control over the one or more shared virtual objects;

providing access to a physics model for each of the one or more shared virtual objects for collision processing;

communicating action and 3D space position data, including collision action data for collisions with the real object for the one or more shared virtual objects to the one or more applications which lack control over the one or more shared virtual objects; and

displaying from a user perspective and by way of the see-through augmented reality display, any of the one or more shared virtual objects having a 3D space position in a display field of view and as having collided with and imparted corresponding virtual-world deformative effects to a corresponding one or more of collision affected parts of the real object, where other parts of the real object which are not collision affected parts can be viewed with use of light passing through corresponding see-through portions of the see-through augmented reality display if not obstructed by virtual objects.

18. The one or more processor readable storage devices of claim 17 further comprising that one or more applications controlling the one or more shared virtual objects are executing on at least one other augmented reality display device system in the user environment.

19. The one or more processor readable storage devices of claim 18 further comprising the one or more applications which lack control over the one or more shared virtual objects designating the one or more shared virtual objects as real objects for control processing.

20. The one or more processor readable storage devices of claim 17 wherein communicating action and 3D space position data for the one or more shared virtual objects to the one or more applications which lack control over the one or more shared virtual objects further comprises:

updating a 3D mapping of a user environment to include space position data of the one or more shared virtual objects.