Extracting a depth map from known camera and model tracking data

• US 20030043270A1
• Filed: 08/29/2001
• Published: 03/06/2003
• Est. Priority Date: 08/29/2001
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Rafey, Richter A., Gibbs, Simon, Wang, Sidney, Broadwell, Peter
• Granted Patent: US 6,940,538 B2
• Time in Patent Office: Days
• US Class Current: 348/157
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method of extracting a depth map from known camera and tracking data is disclosed. The method includes the steps of positioning a virtual camera at the coordinates of the tracked camera, setting the field of view to that of the tracked camera, positioning and orienting a synthetic tracked object to the coordinates of the tracked object, clearing the depth buffer and rendering the tracked object as a depth map.

Claims

1. A method for generating a depth map used to render objects comprising:

establishing a virtual camera using camera data of a tracked camera which defines a viewpoint for a synthetic scene;

setting the field of view of the virtual camera to the field of view of the tracked camera;

positioning a synthetic tracked object in the scene according to position information of the tracked object; and

extracting depth information of the positioned synthetic tracked object to generate depth map.

2. The method as set forth in claim 1, wherein the synthetic scene comprises a three dimensional model of an environment.

3. The method as set forth in claim 1, wherein extracting further comprises extracting depth information of the synthetic scene.

4. The method as set forth in claim 1, wherein the environment comprises a race track.

5. The method as set forth in claim 1, wherein establishing further comprises re-positioning the virtual camera as the tracked camera changes.

6. The method as set forth in claim 1, wherein the camera data is generated by a live camera focused on at least part of a live scene corresponding to the synthetic scene.

7. The method as set forth in claim 1, further comprising animating the synthetic tracked object according to the position and orientation information.

8. The method as set forth in claim 6, further comprising combining the synthetic tracked object with the live scene using depth information stored in the depth map.

9. The method as set forth in claim 8, further comprising using image segmentation to align the depth map in accordance with real edges in the live scene.

10. The method as set forth in claim 1, wherein camera data is recorded from a camera focused on at least part of a scene corresponding to the synthetic scene, the method further comprising combining the synthetic tracked object with video using depth information stored in the depth map.

11. The method as set forth in claim 1, wherein the depth map is further refined by distorting grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.

12. A system for generating a depth map used to render objects comprising:

a tracked camera configured to define a viewpoint for a synthetic scene;

a virtual camera using camera data of the tracked camera, the field of view of the virtual camera set to the field of view of the tracked camera;

a synthetic tracked object in the scene positioned according to position information of the tracked object, wherein depth information of the tracked object provides information to generate a depth map.

13. The system as set forth in claim 12, wherein the synthetic scene comprises a three dimensional model of an environment.

14. The system as set forth in claim 12, wherein the environment comprises a race track.

15. The system as set forth in claim 12, wherein depth information of the synthetic scene further provides information to generate the depth map.

16. The system as set forth in claim 12, wherein the virtual camera is repositioned to track camera data changes.

17. The system as set forth in claim 12, wherein the tracked camera comprises a live camera and the camera data is generated by the tracked camera focused on at least part of a live scene corresponding to the synthetic scene.

18. The system as set forth in claim 12, further comprising an animation unit to animate the model of the tracked object according to the position and orientation information.

19. The system as set forth in claim 17, further comprising a rendering unit configured to generate the synthetic object with the live scene using depth information stored in the depth map.

20. The system as set forth in claim 19, wherein the rendering unit is further configured to use image segmentation to define the depth map in accordance with real edges in the live scene.

21. The system as set forth in claim 20, wherein the rendering unit distorts grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.

22. A system for generating a depth map used to render synthetic objects comprising:

a tracked camera configured to define a viewpoint for a synthetic scene;

a processor configured to generate a virtual camera using camera data of the tracked camera, the field of view of the virtual camera set to the field of view of the tracked camera and a synthetic tracked object in the scene positioned according to position information of the tracked object; and

a depth map of the positioned tracked object.

23. The system as set forth in claim 22, wherein the synthetic scene comprises a three dimensional model of an environment.

24. The system as set forth in claim 22, wherein depth information of the synthetic scene further provides information to generate the depth map.

25. The system as set forth in claim 22, wherein the virtual camera is repositioned to track camera data changes.

26. The system as set forth in claim 22, wherein the tracked camera comprises a live camera coupled to the processor and the camera data is generated by the tracked camera focused on at least part of a live scene corresponding to the synthetic scene.

27. The system as set forth in claim 22,wherein the processor is further configured to animate the model of the tracked object according to the position and orientation information.

28. The system as set forth in claim 26, wherein the processor is further configured to generate the synthetic model with the live scene using depth information stored in the depth map.

29. The system as set forth in claim 28, wherein the processor is further configured to use image segmentation to define the depth map in accordance with real edges in the live scene.

30. The system as set forth in claim 29, wherein the processor is further configured to distort grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.

31. A computer readable medium comprising instructions, which when executed by a processing system perform a method for generating a depth map used to render objects comprising:

establishing a virtual camera using camera data of a tracked camera which defines a viewpoint for a synthetic scene;

setting the field of view of the virtual camera to the field of view of the tracked camera;

positioning a synthetic tracked object in the scene according to position information of the tracked object; and

extracting depth information of the positioned synthetic tracked object to generate a depth map.

32. The computer readable medium as set forth in claim 31, wherein the synthetic scene comprises a three dimensional model of an environment.

33. The computer readable medium as set forth in claim 31, wherein establishing further comprises re-positioning the virtual camera to track camera data changes.

34. The computer readable medium as set forth in claim 31, wherein depth information of the synthetic scene is further used to generate the depth map.

35. The computer readable medium as set forth in claim 31, wherein the camera data is generated by a live camera focused on at least part of a live scene corresponding to the synthetic scene.

36. The computer readable medium as set forth in claim 31, further comprising animating the model of the tracked object according to the position and orientation information.

37. The computer readable medium as set forth in claim 35, further comprising combining the synthetic object with the live scene using depth information stored in the depth map.

38. The computer readable medium as set forth in claim 37, further comprising using image segmentation to define the depth map in accordance with real edges in the live scene.

39. The computer readable medium as set forth in claim 31, further comprising distorting grid coordinates of the depth map based upon characteristics of the tracked camera which defines the field of view.