SYSTEMS AND METHODS FOR SIMULATING THREE-DIMENSIONAL VIRTUAL INTERACTIONS FROM TWO-DIMENSIONAL CAMERA IMAGES

• US 20100194863A1
• Filed: 02/02/2009
• Published: 08/05/2010
• Est. Priority Date: 02/02/2009
• Status: Active Grant

• Current Assignee: YDreams - Informatica S.A.
• Original Assignee: YDreams - Informatica S.A.
• Inventors: Pereira de Almeida, Andre Rui Soares, Frazao, Joao Pedro Gomes de Silva, Vaz de Almada, Antao Bastos Carrico, Lopes, Goncalo Cardoso, Franco, Ivan de Almeida Soares, Cardoso, Nuno Ricardo Sequeira
• Granted Patent: US 8,624,962 B2
• Time in Patent Office: Days
• US Class Current: 348/50
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points

Abstract

A computer implemented method for incorporating a representation of a participant into a virtual 3D environment substantially in real-time is provided. An image including a participant is captured by a camera. A contour of the participant is automatically determined. Depth data is automatically associated with the participant contour. A first virtual 3D representation of the participant is automatically generated by extruding the participant contour based on the associated depth data. An interaction between the first virtual 3D representation of the participant and a second virtual 3D representation of a second object is displayed.

Claims

1. A computer implemented method for incorporating a representation of a participant into a virtual 3D environment, comprising:

receiving an image captured by a camera, the image including a participant;

automatically determining a contour of the participant;

automatically associating depth data with the contour of the participant;

automatically generating a first virtual 3D representation of the participant by extruding the contour based on the associated depth data;

determining an interaction between the first virtual 3D representation of the participant and a second virtual 3D representation of a second object based at least on the extruded contour; and

automatically causing a display of the interaction between the first virtual 3D representation of the participant and the second virtual 3D representation of a second object based at least on the extruded contour,

wherein the computer implemented method is performed substantially in real-time.

2. The computer implemented method of claim 1, further comprising:

the image captured by the camera including multiple participants;

for each of the multiple participants;

automatically determining a contour of the participant;

automatically associating depth data with the contour of the participant; and

automatically generating a virtual 3D representation of the participant by extruding the contour based on the associated depth data.

3. The computer implemented method of claim 1, further comprising:

analyzing the location of a point on the contour of the participant relative to one or more reference points defined by the captured image; and

automatically determining a position of the participant based at least on the analysis of the location the point on the contour of the participant relative to the one or more of reference points defined by the captured image.

4. The computer implemented method of claim 3 further comprising:

receiving camera configuration data indicating at least a position and an orientation of the camera, wherein receiving camera configuration data includes at least one of;

receiving user-determined data via a user input,

receiving data automatically, and

analyzing the image to determine a location of a physical marker in the image; and

determining the position of the participant based at least on (a) the analysis of the location of a point on the contour of the participant relative to one or more reference points defined by the captured image; and

(b) the received camera configuration data.

5. The computer implemented method of claim 1 wherein determining the interaction between the first virtual 3D representation of the participant and a second virtual 3D representation of a second object based at least on the extruded contour is performed at least in part by a real-time physics engine.

6. The computer implemented method of claim 1, further comprising:

mapping at least a portion of the image corresponding to the participant onto the first virtual 3D representation of the participant; and

automatically causing a display of the virtual 3D representation of the participant, which includes the mapped portion of the image corresponding to the participant.

7. The computer implemented method of claim 1 wherein the interaction between the first virtual 3D representation of the participant and the second virtual 3D representation of the second object comprises at least one of:

a shadow cast by the first virtual 3D representation of the participant onto the second virtual 3D representation of the second object;

a shadow cast by the second virtual 3D representation of the second object onto the first virtual 3D representation of the participant;

a collision or contact between the first virtual 3D representation of the participant and the second virtual 3D representation of the second object;

at least a partial occlusion of the first virtual 3D representation of the participant by the second virtual 3D representation of the second object; and

at least a partial occlusion of the second virtual 3D representation of the second object onto the first virtual 3D representation of the participant.

8. The computer implemented method of claim 1 further comprising:

determining a vector representing a viewpoint and a direction of the camera, wherein extruding the contour by an amount corresponding to the associated depth comprises at least one of;

a conical extrusion of uniform depth extending away from the viewpoint of the camera;

an extrusion of uniform depth extending straight-back from the camera;

an extrusion away from the camera and including at least one beveled edge; and

an extrusion of non-uniform depth.

9. Software embodied in tangible computer-readable media and, when executed by a processor, operable to:

receive an image captured by a camera, the image including a participant;

determine a contour of the participant;

associate depth data with the contour of the participant;

generate a first virtual 3D representation of the participant by extruding the contour based on the associated depth data;

determine an interaction between the first virtual 3D representation of the participant and a second virtual 3D representation of a second object based at least on the extruded contour; and

cause a display of the interaction between the first virtual 3D representation of the participant and the second virtual 3D representation of a second object based at least on the extruded contour;

wherein the computer software is operable to run substantially in real-time.

10. The software of claim 9 further operable to:

analyze the location of a point on the contour of the participant relative to one or more reference points defined by the captured image; and

automatically determine a position of the participant based at least on the analysis of the location the point on the contour of the participant relative to the one or more of reference points defined by the captured image.

11. The software of claim 10 further operable to:

receive camera configuration data indicating a position and an orientation of the camera, wherein the camera configuration data includes at least one of;

user entered user-determined data,

automatically determined data, and

data determined by analyzing the image to determine a location of a physical marker in the image; and

determine the position of the participant based at least on (a) the analysis of the location of a point on the contour of the participant relative to one or more reference points defined by the captured image; and

(b) the received camera configuration data.

12. The software of claim 9 operable to determine the interaction between the first virtual 3D representation of the participant and a second virtual 3D representation of a second object based at least on the extruded contour using, at least in part, a real-time physics engine.

13. The software of claim 9, further operable to:

map at least a portion of the image corresponding to the participant onto the first virtual 3D representation of the participant; and

automatically cause a display of the virtual 3D representation of the participant includes the mapped portion of the image corresponding to the participant.

14. The software of claim 9 wherein the interaction between the first virtual 3D representation of the participant and the second virtual 3D representation of the second object based at least on the extruded contour comprises at least one of:

a shadow cast by the first virtual 3D representation of the participant onto the second virtual 3D representation of the second object;

a shadow cast by the second virtual 3D representation of the second object onto the first virtual 3D representation of the participant;

a collision or contact between the first virtual 3D representation of the participant and the second virtual 3D representation of the second object;

at least a partial occlusion of the first virtual 3D representation of the participant by the second virtual 3D representation of the second object; and

at least a partial occlusion of the second virtual 3D representation of the second object onto the first virtual 3D representation of the participant.

15. The software of claim 9 further operable to:

determine a vector representing a viewpoint and a direction of the camera, wherein the extrusion of the contour by an amount corresponding to the associated depth comprises at least one of;

a conical extrusion of uniform depth extending away from the viewpoint of the camera;

an extrusion of uniform depth extending straight-back from the camera;

a conical extrusion of uniform depth wherein the exterior edges of the conical extrusion that are furthest from the contour are beveled, thereby reducing the volume of the conical extrusion; and

an extrusion of non-uniform depth.

19. The computing system of claim 15 enabled to determine the interaction between the first virtual 3D representation of the participant and a second virtual 3D representation of a second object based at least on the extruded contour using, at least in part, a real-time physics engine.

20. The computing system of claim 15, further enabled to:

map at least a portion of the image corresponding to the participant onto the first virtual 3D representation of the participant; and

automatically cause a display of the virtual 3D representation of the participant includes the mapped portion of the image corresponding to the participant.

16. A computing system comprising:

a processor;

a memory coupled to the processor; and

an interactive media subsystem enabled to;

receive an image captured by a camera, the image including a participant;

automatically determine a contour of the participant;

automatically associate depth data with the contour of the participant;

automatically generate a first virtual 3D representation of the participant by extruding the contour based on the associated depth data;

determine an interaction between the first virtual 3D representation of the participant and a second virtual 3D representation of a second object based at least on the extruded contour; and

automatically cause a display of the interaction between the first virtual 3D representation of the participant and the second virtual 3D representation of a second object based at least on the extruded contour;

wherein the interactive media subsystem operates substantially in real-time.

17. The computing system of claim 16, wherein the interactive media subsystem is further enabled to:

analyze the location of a point on the contour of the participant relative to one or more reference points defined by the captured image; and

automatically determine a position of the participant based at least on the analysis of the location the point on the contour of the participant relative to the one or more of reference points defined by the captured image

18. The computing system of claim 17 further operable to:

receive camera configuration data indicating a position and an orientation of the camera, wherein the camera configuration data includes at least one of;

user entered user-determined data,

automatically determined data, and

data determined by analyzing the image to determine a location of a physical marker in the image; and

determine the position of the participant based at least on (a) the analysis of the location of a point on the contour of the participant relative to one or more reference points defined by the captured image; and

(b) the received camera configuration data.