Extracting a depth map from known camera and model tracking data

• US 6,940,538 B2
• Filed: 08/29/2001
• Issued: 09/06/2005
• Est. Priority Date: 08/29/2001
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Rafey, Richter A., Gibbs, Simon, Wang, Sidney, Broadwell, Peter
• Primary Examiner(s): Le, Vu
• Application Number:
  US09/942,806
• Publication Number:
  US 20030043270A1
• Time in Patent Office: 1,469 Days
• Field of Search: 348/46, 348/42, 348/51, 348/580, 348/157, 382/154, 382/285, 345/419, 345/757, 345/848, 345/850, 345/629, 463 5- 6
• US Class Current: 348/157
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method of extracting a depth map from known camera and tracking data. The method includes the steps of positioning a virtual camera at the coordinates of the tracked camera, setting the field of view to that of the tracked camera, positioning and orienting a synthetic tracked object to the coordinates of the tracked object, clearing the depth buffer and rendering the tracked object as a depth map.

Claims

1. A method for generating a depth map used to render objects comprising:

establishing a virtual camera using camera data of a tracked camera which defines a viewpoint for a synthetic scene, wherein the camera data is generated by a live camera focused on at least part of a live scene corresponding to the synthetic scene;

setting the field of view of the virtual camera to the field of view of the tracked camera;

positioning a synthetic tracked object in the scene according to position information of a tracked object;

embedding the synthetic tracked object into the live scene using depth information stored in the depth map; and

extracting depth information of the positioned synthetic tracked object to generate depth map.

2. The method as set forth in claim 1, wherein the synthetic scene comprises a three dimensional model of an environment.

3. The method as set forth in claim 1, wherein extracting further comprises extracting depth information of the synthetic scene.

4. The method as set forth in claim 1, wherein the environment comprises a race track.

5. The method as set forth in claim 1, wherein establishing further comprises re-positioning the virtual camera as the tracked camera changes.

6. The method as set forth in claim 1, further comprising animating the synthetic tracked object according to the position and orientation information.

7. The method as set forth in claim 1, further comprising using image segmentation to align the depth map in accordance with real edges in the live scene.

8. The method as set forth in claim 1, wherein camera data is recorded from a camera focused on at least part of a scene corresponding to the synthetic scene, the method further comprising combining the synthetic tracked object with video using depth information stored in the depth map.

9. The method as set forth in claim 1, wherein the depth map is further refined by distorting grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.

10. A system comprising:

a tracked camera configured to define a viewpoint for a synthetic scene, wherein the tracked camera comprises a live camera and the camera data is generated by the tracked camera focused on at least part of a live scene corresponding to the synthetic scene;

a virtual camera using camera data of the tracked camera, the field of view of the virtual camera set to the field of view of the tracked camera;

a synthetic tracked object in the scene positioned according to position information of a tracked object, wherein depth information of the tracked object provides information to generate a depth man; and

a rendering unit configured to embed the synthetic tracked object into the live scene using depth information stored in the depth map.

11. The system as set forth in claim 10, wherein the synthetic scene comprises a three dimensional model of an environment.

12. The system as set forth in claim 10, wherein the environment comprises a race track.

13. The system as set forth in claim 10, wherein depth information of the synthetic scene further provides information to generate the depth map.

14. The system as set forth in claim 10, wherein the virtual camera is re-positioned to track camera data changes.

15. The system as set forth in claim 10, further comprising an animation unit to animate the synthetic tracked object according to the position and orientation information.

16. The system as set forth in claim 10, wherein the rendering unit is further configured to use image segmentation to define the depth map in accordance with real edges in the live scene.

17. The system as set forth in claim 16, wherein the rendering unit distorts grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.

18. The system as set forth in claim 10, wherein the synthetic scene comprises a three dimensional model of an environment.

19. The system as set forth in claim 10, wherein depth information of the synthetic scene further provides information to generate the depth map.

20. The system as set forth in claim 10, wherein the virtual camera is re-positioned to track camera data changes.

21. A system for generating a depth map used to render synthetic objects comprising:

a tracked camera configured to define a viewpoint for a synthetic scene, wherein the tracked camera comprises a live camera coupled to the processor and the camera data is generated by the tracked camera focused on at least part of a live scene corresponding to the synthetic scene;

a processor configured to generate a virtual camera using camera data of the tracked camera, the field of view of the virtual camera set to the field of view of the tracked camera and a synthetic tracked object in the scene positioned according to position information of the tracked object, wherein the processor is further configured to embed the synthetic tracked object into the live scene using depth information stored in the depth map; and

a depth map of the positioned tracked object.

22. The system as set forth in claim 21, wherein the processor is further configured to animate the tracked synthetic object according to the position and orientation information.

23. The system as set forth in claim 21, wherein the processor is further configured to use image segmentation to define the depth map in accordance with real edges in the live scene.

24. The system as set forth in claim 23, wherein the processor is further configured to distort grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.

25. A computer readable medium comprising instructions, which when executed by a processing system perform a method for generating a depth map used to render objects comprising:

establishing a virtual camera using camera data of a tracked camera which defines a viewpoint for a synthetic scene, wherein the camera data is generated by a live camera focused on at least part of a live scene corresponding to the synthetic scene;

setting the field of view of the virtual camera to the field of view of the tracked camera;

positioning a synthetic tracked object in the scene according to position information of the tracked object;

extracting depth information of the positioned synthetic tracked object to generate a depth map; and

embedding the synthetic tracked object into the live scene using depth information stored in the depth map.

26. The computer readable medium as set forth in claim 25, wherein the synthetic scene comprises a three dimensional model of an environment.

27. The computer readable medium as set forth in claim 25, wherein establishing further comprises re-positioning the virtual camera to track camera data changes.

28. The computer readable medium as set forth in claim 25, wherein depth information of the synthetic scene is further used to generate the depth map.

29. The computer readable medium as set forth in claim 25, further comprising animating the tracked synthetic object according to the position and orientation information.

30. The computer readable medium as set forth in claim 25, further comprising using image segmentation to define the depth map in accordance with real edges in the live scene.

31. The computer readable medium as set forth in claim 25, further comprising distorting grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.