System and method for providing a real-time three-dimensional interactive environment

• US 20040239670A1
• Filed: 05/29/2003
• Published: 12/02/2004
• Est. Priority Date: 05/29/2003
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard
• Granted Patent: US 8,072,470 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An invention is provided for affording a real-time three-dimensional interactive environment using a three-dimensional camera. The invention includes obtaining two-dimensional data values for a plurality of pixels representing a physical scene, and obtaining a depth value for each pixel of the plurality of pixels using a depth sensing device. Each depth value indicates a distance from a physical object in the physical scene to the depth sensing device. At least one computer-generated virtual object is inserted into the scene, and an interaction between a physical object in the scene and the virtual object is detected based on coordinates of the virtual object and the obtained depth values.

Claims

1. A method for providing a real-time three-dimensional interactive environment, comprising the operations of:

obtaining two-dimensional data values for a plurality of pixels representing a physical scene;

obtaining a depth value for each pixel of the plurality of pixels using a depth sensing device, each depth value indicating a distance from a physical object in the physical scene to the depth sensing device;

inserting at least one virtual object into the scene, the virtual object being computer-generated; and

detecting an interaction between a physical object in the scene and the virtual object based on coordinates of the virtual object and the obtained depth values.

2. A method as recited in claim 1, wherein the two-dimensional values for the plurality of pixels are color values.

3. A method as recited in claim 1, wherein each depth value indicates a distance from a physical object in the physical scene represented by the corresponding pixel to the sensing device.

4. A method as recited in claim 1, wherein the interaction is a collision between a physical object in the scene and the virtual object.

5. A method as recited in claim 4, further comprising the operation of detecting when the virtual object and a physical object occupy a same three-dimensional space based on three-dimensional coordinates of the virtual object and three-dimensional coordinates of the physical object.

6. A method as recited in claim 1, further comprising the operation of visually altering an appearance of a physical object in the scene.

7. A method as recited in claim 6, wherein the physical object is a user.

8. A method as recited in claim 7, wherein computer-generated clothing is mapped to the user based on the depth values for pixels representing the user.

9. A method as recited in claim 1, further comprising the operation of defining a maximum depth range, the maximum depth range indicating the farthest distance for which depth values will be obtained.

10. A method as recited in claim 9, further comprising the operation detecting depth values for a user only when the user is within a distance less than the maximum depth range to the sensing device.

11. A method as recited in claim 1, wherein the depth sensing device is a depth camera.

12. A computer program embodied on a computer readable medium for providing a real-time three-dimensional interactive environment, comprising:

program instructions that obtain two-dimensional data values for a plurality of pixels representing a physical scene;

program instructions that obtain a depth value for each pixel of the plurality of pixels using a depth sensing device, each depth value indicating a distance from a physical object in the physical scene to the depth sensing device;

program instructions that insert at least one virtual object into the scene, the virtual object being computer-generated; and

program instructions that detect an interaction between a physical object in the scene and the virtual object based on coordinates of the virtual object and the obtained depth values.

13. A computer program as recited in claim 12, wherein the two-dimensional values for the plurality of pixels are color values.

14. A computer program as recited in claim 12, wherein each depth value indicates a distance from a physical object in the physical scene represented by the corresponding pixel to the sensing device.

15. A computer program as recited in claim 12, wherein the interaction is a collision between a physical object in the scene and the virtual object.

16. A computer program as recited in claim 12, further comprising program instructions that define a maximum depth range, the maximum depth range indicating the farthest distance for which depth values will be obtained.

17. A system for providing a real-time three-dimensional interactive environment, comprising:

a depth sensing device capable of obtaining two-dimensional data values for a plurality of pixels representing a physical scene, wherein the depth sensing device is further capable of obtaining a depth value for each pixel of the plurality of pixels, each depth value indicating a distance from a physical object in the physical scene to the depth sensing device;

logic that inserts at least one virtual object into the scene, the virtual object being computer-generated; and

logic that detects an interaction between a physical object in the scene and the virtual object based on coordinates of the virtual object and the obtained depth values.

18. A system as recited in claim 17, wherein the two-dimensional values for the plurality of pixels are color values, and wherein each depth value indicates a distance from a physical object in the physical scene represented by the corresponding pixel to the sensing device.

19. A system as recited in claim 17, further comprising logic that defines a maximum depth range, the maximum depth range indicating the farthest distance for which depth values will be obtained.

20. A system as recited in claim 19, further comprising logic that detects depth values for a user only when the user is within a distance less than the maximum depth range to the sensing device.

21. A method for providing a real-time three-dimensional interactive environment, comprising the operations of:

obtaining two-dimensional data values for a plurality of pixels representing a physical scene;

obtaining a depth value for each pixel of the plurality of pixels using a depth sensing device, each depth value indicating a distance from a physical object in the physical scene to the depth sensing device; and

estimating three-dimensional volume information for each physical object in the physical scene based on the obtained two-dimensional data values and the obtained depth values.

22. A method as recited in claim 21, further comprising the operation of inserting at least one virtual object into the scene, the virtual object being computer-generated, wherein the virtual object includes three-dimensional volume information for the virtual object.

23. A method as recited in claim 22, further comprising the operation of detecting an interaction between a physical object in the scene and the virtual object based on coordinates of the three-dimensional volume information for the virtual object and the three-dimensional volume information for the physical object.

24. A method as recited in claim 21, wherein the two-dimensional values for the plurality of pixels are color values.

25. A method as recited in claim 21, wherein each depth value indicates a distance from a physical object in the physical scene represented by the corresponding pixel to the sensing device.

26. A computer program embodied on a computer readable medium for providing a real-time three-dimensional interactive environment, comprising:

program instructions that obtain two-dimensional data values for a plurality of pixels representing a physical scene;

program instructions that obtain a depth value for each pixel of the plurality of pixels using a depth sensing device, each depth value indicating a distance from a physical object in the physical scene to the depth sensing device; and

program instructions that estimate three-dimensional volume information for each physical object in the physical scene based on the obtained two-dimensional data values and the obtained depth values.

27. A computer program as recited in claim 26, further comprising program instructions that insert at least one virtual object into the scene, the virtual object being computer-generated, wherein the virtual object includes three-dimensional volume information for the virtual object.

28. A computer program as recited in claim 27, further comprising program instructions that detect an interaction between a physical object in the scene and the virtual object based on coordinates of the three-dimensional volume information for the virtual object and the three-dimensional volume information for the physical object.

29. A computer program as recited in claim 26, wherein the two-dimensional values for the plurality of pixels are color values.

30. A computer program as recited in claim 26, wherein each depth value indicates a distance from a physical object in the physical scene represented by the corresponding pixel to the sensing device.