Methods and Systems for Performing 3D Simulation Based on a 2D Video Image

• US 20200134911A1
• Filed: 10/28/2019
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: van Hoff, Arthur, Kopeinigg, Daniel, Lee, Philip, Hajmohammadi, Solmaz, Khire, Sourabh, Venshtain, Simion
• Granted Patent: US 11,217,006 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/292 Multi-camera tracking

Abstract

An exemplary three-dimensional (3D) simulation system accesses a two-dimensional (2D) video image captured by a video capture device and that depicts a bounded real-world scene and a real-world object present within the bounded real-world scene. The 3D simulation system accesses respective 3D models of the bounded real-world scene and the real-world object. Based on the 2D video image, the 3D simulation system tracks a spatial characteristic of the real-world object relative to the bounded real-world scene. Based on the tracked spatial characteristic of the real-world object and the 3D models of the bounded real-world scene and the real-world object, the 3D simulation system generates a 3D simulation of the bounded real-world scene within which the real-world object is simulated in accordance with the tracked spatial characteristic of the real-world object. Corresponding methods and systems are also disclosed.

Claims

1. A method comprising:

accessing, by a three-dimensional (3D) simulation system, a two-dimensional (2D) video image captured by a video capture device and depicting a bounded real-world scene and a real-world object present within the bounded real-world scene;

accessing, by the 3D simulation system, a 3D model of the bounded real-world scene and a 3D model of the real-world object;

tracking, by the 3D simulation system based on the 2D video image, a spatial characteristic of the real-world object relative to the bounded real-world scene; and

generating, by the 3D simulation system based on the tracked spatial characteristic of the real-world object and the 3D models of the bounded real-world scene and the real-world object, a 3D simulation of the bounded real-world scene within which the real-world object is simulated in accordance with the tracked spatial characteristic of the real-world object.

2. The method of claim 1, wherein the 3D models of the bounded real-world scene and the real-world object are generated prior to the accessing of the 2D video image and are stored in a storage repository accessible to the 3D simulation system.

3. The method of claim 1, further comprising:

identifying, by the 3D simulation system prior to the accessing of the 3D models, a fixed space within which a real-world event is confined;

defining, by the 3D simulation system based on the identified fixed space, a boundary of the bounded real-world scene; and

generating, by the 3D simulation system based on the defined boundary, the 3D model of the bounded real-world scene.

4. The method of claim 1, wherein:

the 2D video image is represented by color data that depicts the bounded real-world scene and the real-world object; and

the tracking of the spatial characteristic is performed based exclusively on a set of data including the color data representing the 2D video image and excluding any depth data representative of the bounded real-world scene or the real-world object.

5. The method of claim 1, wherein:

the method further comprises accessing, by the 3D simulation system, additional 2D video images associated with the 2D video image and captured by additional video capture devices disposed at different vantage points in relation to the bounded real-world scene;

the 2D video image and the additional 2D video images each depict the bounded real-world scene and the real-world object using color data and depth data representative of the bounded real-world scene and the real-world object; and

the accessing of the 3D models of the bounded real-world scene and the real-world object is performed by generating the 3D models based on the color data and the depth data from the 2D video image and the additional 2D video images.

6. The method of claim 1, wherein the spatial characteristic of the real-world object that is tracked relative to the bounded real-world scene is a pose of the real-world object, the pose comprising a combination of a spatial orientation of the real-world object and a spatial location of the real-world object.

7. The method of claim 6, wherein:

the real-world object is a person present within the bounded real-world scene, the person having a plurality of body parts including a particular body part;

the 3D simulation system employs machine learning technology that has been trained based on a plurality of events that are similar to, and that have occurred prior to, an event associated with the 2D video image; and

the tracking of the pose is performed using the machine learning technology and includes at least one of;

tracking the particular body part for a frame of the 2D video image within which the particular body part is not depicted due to being occluded from a vantage point of the video capture device, and

differentiating the person from a body part of a different person present within the bounded real-world scene, the body part of the different person contiguous with the depiction of the person within a frame of the 2D video image due to being colinear with the person and a vantage point of the video capture device.

8. The method of claim 1, further comprising predicting, by the 3D simulation system at a first time and based on the 2D video image, the spatial characteristic of the real-world object as the spatial characteristic will be detected at a second time that occurs after the first time;

wherein the tracking of the spatial characteristic performed at the second time is based on the predicting of the spatial characteristic that was performed at the first time.

9. The method of claim 1, wherein:

the 2D video image captured by the video capture device depicts a plurality of real-world objects each of which is present within the bounded real-world scene and one of which is the real-world object; and

the plurality of real-world objects includes a person and an inanimate object that is used by the person.

10. The method of claim 1, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

11. A method comprising:

receiving, by a three-dimensional (3D) simulation system, a televised two-dimensional (2D) video image captured by a television camera and depicting

a bounded real-world scene within which a televised real-world event is occurring, and

a person associated with the televised real-world event and present within the bounded real-world scene;

accessing, by the 3D simulation system, a 3D model of the bounded real-world scene and a 3D model of the person associated with the televised real-world event;

tracking, by the 3D simulation system based on the televised 2D video image, a pose of the person relative to the bounded real-world scene, the pose comprising a combination of a spatial orientation of the person and a spatial location of the person; and

presenting, by the 3D simulation system as an augmentation viewable by way of an augmented reality media player device, a 3D simulation of the bounded real-world scene within which the person is simulated in accordance with the tracked pose, the 3D simulation generated based on the tracked pose of the person and the 3D models of the bounded real-world scene and the person, and the augmentation configured to be presented in a vicinity of a display screen that is presenting the televised real-world event.

12. The method of claim 11, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

13. A system comprising:

a memory storing instructions; and

a processor communicatively coupled to the memory and configured to execute the instructions to;

access a two-dimensional (2D) video image captured by a video capture device and depicting a bounded real-world scene and a real-world object present within the bounded real-world scene;

access a 3D model of the bounded real-world scene and a 3D model of the real-world object;

track, based on the 2D video image, a spatial characteristic of the real-world object relative to the bounded real-world scene; and

generate, based on the tracked spatial characteristic of the real-world object and the 3D models of the bounded real-world scene and the real-world object, a 3D simulation of the bounded real-world scene within which the real-world object is simulated in accordance with the tracked spatial characteristic of the real-world object.

14. The system of claim 13, wherein the 3D models of the bounded real-world scene and the real-world object are generated prior to the accessing of the 2D video image and are stored in a storage repository accessible to the system.

15. The system of claim 13, wherein the processor is further configured to execute the instructions to:

identify, prior to the accessing of the 3D models, a fixed space within which a real-world event is confined;

define, based on the identified fixed space, a boundary of the bounded real-world scene; and

generate, based on the defined boundary, the 3D model of the bounded real-world scene.

16. The system of claim 13, wherein:

the 2D video image is represented by color data that depicts the bounded real-world scene and the real-world object; and

the tracking of the spatial characteristic is performed based exclusively on a set of data including the color data representing the 2D video image and excluding any depth data representative of the bounded real-world scene or the real-world object.

17. The system of claim 13, wherein:

the processor is further configured to execute the instructions to access additional 2D video images associated with the 2D video image and captured by additional video capture devices disposed at different vantage points in relation to the bounded real-world scene;

the 2D video image and the additional 2D video images each depict the bounded real-world scene and the real-world object using color data and depth data representative of the bounded real-world scene and the real-world object; and

the accessing of the 3D models of the bounded real-world scene and the real-world object is performed by generating the 3D models based on the color data and the depth data from the 2D video image and the additional 2D video images.

18. The system of claim 13, wherein the spatial characteristic of the real-world object that is tracked relative to the bounded real-world scene is a pose of the real-world object, the pose comprising a combination of a spatial orientation of the real-world object and a spatial location of the real-world object.

19. The system of claim 18, wherein:

the real-world object is a person present within the bounded real-world scene, the person having a plurality of body parts including a particular body part;

the system employs machine learning technology that has been trained based on a plurality of events that are similar to, and that have occurred prior to, an event associated with the 2D video image; and

the tracking of the pose is performed using the machine learning technology and includes at least one of;

tracking the particular body part for a frame of the 2D video image within which the particular body part is not depicted due to being occluded from a vantage point of the video capture device, and

differentiating the person from a body part of a different person present within the bounded real-world scene, the body part of the different person contiguous with the depiction of the person within a frame of the 2D video image due to being colinear with the person and a vantage point of the video capture device.

20. The system of claim 13, wherein the processor is further configured to execute the instructions to predict, at a first time and based on the 2D video image, the spatial characteristic of the real-world object as the spatial characteristic will be detected at a second time that occurs after the first time;

wherein the tracking of the spatial characteristic performed at the second time is based on the predicting of the spatial characteristic that was performed at the first time.