Visualization of physical interactions in augmented reality

• US 9,558,592 B2
• Filed: 08/15/2014
• Issued: 01/31/2017
• Est. Priority Date: 12/31/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew, Broaddus, Christopher
• Primary Examiner(s): He, Yingchun
• Application Number:
  US14/461,257
• Publication Number:
  US 20160049005A1
• Time in Patent Office: 900 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/21 Collision detection, intersection

Abstract

A system and method for visualization of physical interactions are described. Objects in a scene are captured with a viewing device. Physical characteristics of the objects are computed using data from at least one sensor corresponding to the objects. A physics model of predicted interactions between the one or more objects is generated using the physical characteristics of the objects. An interaction visualization is generated based on the physics model of the predicted interactions between the one or more objects. An image of the one or more objects is augmented with the interaction visualization in a display of the viewing device.

Claims

1. A viewing device comprising:

a camera;

a processor comprising an augmented reality application, the augmented reality application configured to;

identify two or more objects in a scene captured with the camera of the viewing device;

identify a plurality of three-dimensional simulated trajectory paths within the scene based on a same source related to a first object in the scene;

identify areas within the scene that are not affected by the plurality of three-dimensional simulated trajectory paths;

compute physical characteristics of the two or more objects using data from at least one sensor corresponding to the two or more objects;

generate a physics model of predicted interactions between the two or more objects using the physical characteristics of the two or more objects;

generate an interaction visualization based on the physics model of the predicted interactions between the two or more objects; and

augment an image of the two or more objects with the interaction visualization in a display of the viewing device;

identify a second object that can withstand an impact from the first object using physical characteristics of the first object and the second objects; and

augment an image of the second object in the display of the viewing device.

2. The viewing device of claim 1, wherein the augmented reality application is further configured to:

generate a characteristics visualization of the physical characteristics of the two or more objects; and

augment the image of the two or more objects with the characteristics visualization of the two or more objects in the display of the viewing device,

wherein the data comprises a live data stream from the at least one sensor.

3. The viewing device of claim 1, wherein the augmented reality application is configured to:

identify an area in the scene affected by the predicted interactions between the two or more objects;

render a virtual object corresponding to the area affected by the predicted interactions; and

augment an image of the scene with the virtual object at the identified area.

4. The viewing device of claim 1, wherein the augmented reality application is configured to:

identify an area in the scene unaffected by the predicted interactions between the two or more objects;

render a virtual object corresponding to the area unaffected by the predicted interactions in the scene; and

augment an image of the scene with the virtual object at the identified area.

5. The viewing device of claim 1, wherein the augmented reality application is configured to:

generate a physics model for each identified object using the physical characteristics of the corresponding identified object; and

generate the physics model of predicted interactions using the physics model of each identified object.

6. The viewing device of claim 5, wherein the augmented reality application is configured to:

generate a predicted trajectory of each identified object within the scene based on the physics model for each identified object;

identify a predicted collision and a predicted collision location of the two or more identified objects based on the predicted trajectories within the scene; and

identify an intensity of the predicted collision.

7. The viewing device of claim 1, wherein the augmented reality application is configured to:

generate a physics model of a dynamic object from the two or more objects;

generate a physics model of a static object from the two or more objects; and

generate a physics model of a physical interaction between the dynamic object and the static object,

wherein the interaction visualization module is configured to identify areas in the scene affected by the physical interaction between the dynamic object and the static object, and to augment an image of the identified areas in the scene.

8. The viewing device of claim 1, wherein the augmented reality application is configured to:

generate a physics model of a first and a second dynamic object from the two or more objects;

generate a physics model of a physical interaction between the first and second dynamic objects,

wherein the interaction visualization module is configured to identify areas in the scene affected by the physical interaction between the first and second dynamic objects, and to augment an image of the identified areas in the scene.

9. The viewing device of claim 1, wherein the augmented reality application is configured to:

capture the image of the two or more objects in the scene; and

display the interaction visualization of the two or more objects, a position and size of the interaction visualization of the two or more objects in the transparent display based on a position and orientation of the viewing device relative to the two or more objects.

10. A method comprising:

capturing a scene with a camera of a viewing device;

identifying two or more objects in the scene;

identifying a plurality of three-dimensional simulated trajectory paths within the scene based on a same source related to a first object in the scene;

identifying areas within the scene that are not affected by the plurality of three-dimensional simulated trajectory paths;

computing physical characteristics of the two or more objects using data from at least one sensor corresponding to the two or more objects;

generating a physics model of predicted interactions between the two or more objects using the physical characteristics of the two or more objects;

generating, using a hardware processor of a machine, an interaction visualization based on the physics model of the predicted interactions between the two or more objects;

augmenting an image of the two or more objects with the interaction visualization in a display of the viewing device;

identifying a second object that can withstand an impact from the first object using physical characteristics of the first object and the second objects;

augmenting an image of the second object in the display of the viewing device; and

displaying the augmented image in the display of the viewing device.

11. The method of claim 10, further comprising:

generating a characteristics visualization of the physical characteristics of the two or more objects and augmenting the image of the two or more objects with the characteristics visualization of the two or more objects in the display of the viewing device, wherein the data comprises a live data stream from the at least one sensor.

12. The method of claim 10, further comprising:

identifying an area in the scene affected by the predicted interactions between the two or more objects;

rendering a virtual object corresponding to the area affected by the predicted interactions; and

augmenting an image of the scene with the virtual object at the identified area.

13. The method of claim 10, further comprising:

identifying an area in the scene unaffected by the predicted interactions between the two or more objects;

rendering a virtual object corresponding to the area unaffected by the predicted interactions in the scene; and

augmenting an image of the scene with the virtual object at the identified area.

14. The method of claim 10, further comprising:

generating a physics model for each identified object using the physical characteristics of a corresponding identified object; and

generating the physics model of predicted interactions using the physics model of each identified object.

15. The method of claim 14, further comprising:

generating a predicted trajectory of each identified object within the scene based on the physics model for each identified object;

identifying a predicted collision and a predicted collision location of the two or more identified objects based on the predicted trajectories within the scene; and

identifying an intensity of the predicted collision.

16. The method of claim 10, further comprising:

generating a physics model of a dynamic object from the two or more objects;

generating a physics model of a static object from the two or more objects;

generating a physics model of a physical interaction between the dynamic object and the static object;

identifying areas in the scene affected by the physical interaction between the dynamic object and the static object; and

augmenting an image of the identified areas in the scene.

17. The method of claim 10, further comprising:

generating a physics model of a first and a second dynamic object from the two or more objects;

generating a physics model of a physical interaction between the first and second dynamic objects;

identifying areas in the scene affected by the physical interaction between the first and second dynamic objects; and

augmenting an image of the identified areas in the scene.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

identifying two or more objects in a scene captured with a viewing device;

identifying a plurality of three-dimensional simulated trajectory paths within the scene based on a same source related to a first object in the scene;

identifying areas within the scene that are not affected by the plurality of three-dimensional simulated trajectory paths;

computing physical characteristics of the two or more objects using data from at least one sensor corresponding to the two or more objects;

generating a physics model of predicted interactions between the two or more objects using the physical characteristics of the two or more objects;

generating an interaction visualization based on the physics model of the predicted interactions between the two or more objects;

augmenting an image of the two or more objects with the interaction visualization in a display of the viewing device;

identifying a second object that can withstand an impact from the first object using physical characteristics of the first object and the second objects; and

augmenting an image of the second object in the display of the viewing device.