Enhancing broadcast of an event with synthetic scene using a depth map

• US 6,990,681 B2
• Filed: 03/29/2002
• Issued: 01/24/2006
• Est. Priority Date: 08/09/2001
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Rafey, Richter A., Gibbs, Simon, Wang, Sidney, Gong, Hubert Le Van, Broadwell, Peter
• Primary Examiner(s): An, Shawn S.
• Application Number:
  US10/113,707
• Publication Number:
  US 20030038892A1
• Time in Patent Office: 1,397 Days
• Field of Search: 348/157, 348/143, 348/148, 348/155, 348/169, 348/584, 345/619, 345/419, 725/105, 375/240.01, 375/240.25, 375/240.26
• US Class Current: 725/105
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8017 Driving on land or water; Flying
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A broadcast of an event is enhanced with synthetic scenes generated from audio visual and supplemental data received in the broadcast. A synthetic scene is integrated into the broadcast in accordance with a depth map that contains depth information for the synthetic scene. The supplemental data may be sensing data from various sensors placed at the event, position and orientation data of particular objects at the event, or environmental data on conditions at the event. The supplemental data may also be camera tracking data from a camera that is used to generate a virtual camera and viewpoints for the synthetic scene.

The present invention describes systems, clients, servers, methods, and computer-readable media of varying scope. In addition to the aspects of the present invention described in this summary, further aspects of the invention will become apparent by reference to the drawings and by reading the detailed description that follows.

Claims

1. A method for enhancing a broadcast of an event, comprising:

generating a synthetic scene based on audio visual data and supplemental data received in the broadcast;

generating a depth map to store depth information for the synthetic scene; and

integrating the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises;

establishing the virtual camera using camera tracking data of a tracked camera which defines a viewpoint for the synthetic scene;

setting a field of view of the virtual camera to a corresponding field of view of the tracked camera;

positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object;

extracting depth information of the synthetic tracked object to generate the depth map; and

refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.

2. The method of claim 1, wherein the supplemental data comprises sensing data from a plurality of sensors placed at strategic points to render realistic synthetic movement of a synthetic object in the synthetic scene.

3. The method of claim 1, wherein the supplemental data comprises position and orientation data of an object from a global positioning system receiver, the position and orientation data indicating the position and orientation of the object in a three dimensional space.

4. The method of claim 3, wherein the global positioning system receiver further provides environmental conditions of the event to synchronize with the broadcast.

5. The method of claim 1, wherein the supplemental data comprises camera tracking data from a tracked camera located at the event, the camera tracking data including positions of the tracked camera in a three dimensional space to generate a virtual camera corresponding to the tracked camera.

6. The method of claim 5, further comprising registering the camera tracking data with the broadcast to render a virtual version of an object in the broadcast.

7. The method of claim 1, further comprising a signal processor for processing the audio visual data and the supplemental data to generate the synthetic scene using the depth map.

8. The method of claim 7, further comprising selecting a desired synthetic camera view in response to receiving an input through a user interface.

9. The method of claim 1, further comprising repositioning the virtual camera in response to position changes of the tracked camera.

10. The method of claim 1, further comprising:

reconstructing a virtual view of a model, using the position information and camera tracking data; and

extracting the depth information from the virtual view to generate the depth map.

11. A method for enhancing a broadcast of an event, comprising:

collecting, at a broadcast server, audio visual data and supplemental data from the event;

transmitting the audio visual data and the supplemented data to a broadcast client over a network;

generating, at the broadcast client, a synthetic scene based on the audio visual data and the supplemental data;

generating a depth map to store depth information for the synthetic scene; and

integrating the synthetic scene into the broadcast using the depth map, wherein the generating the depth map comprises;

establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene;

setting a field of view of the virtual camera to corresponding field of view of the tracked camera;

positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object;

extracting depth information of the synthetic tracked object to generate the depth map; and

refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.

12. The method of claim 11, further comprising:

encoding, at the broadcast server, the audio visual data and the supplemental data as motion picture expert group compatible data; and

decoding, at the broadcast client, the motion picture expert group compatible data to retrieve the audio visual data and supplemental data.

13. The method of claim 11, wherein the supplemental data comprises:

sensing data from a plurality of sensors placed at the strategic points of the event;

position and orientation data from a global positioning system receiver; and

camera tracking data from a tracked camera.

14. The method of claim 11, further comprising selecting a desired synthetic camera view in response to receiving an input at the broadcast client.

15. A system for enhancing a broadcast of an event, comprising:

a video signal unit coupled to provide audio visual data from the event;

a supplemental data unit coupled to provide supplemental data from the event;

a depth map coupled to provide depth information; and

a processing unit configured to process the audio visual data and the supplemental data to generate a synthetic scene and further, the processing unit configured to integrate the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises;

establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene;

setting a field of view of the virtual camera to corresponding field of view of the tracked camera;

positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object;

extracting depth information of the synthetic tracked object to generate the depth map; and

refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.

16. The system of claim 15, further comprising a viewer control unit to receive a user selection of the synthetic scene.

17. The system of claim 15, further comprising:

a plurality of sensors placed at strategic points of a synthetic object to provide sensing data to render realistic synthetic movement of the synthetic object in the synthetic scene;

a global positioning system receiver to provide position and orientation data of the synthetic object, the position and orientation data indicating the position and orientation of the synthetic object in a three dimensional space; and

a tracked camera located at the event to provide camera track data, the camera track data including positions of the tracked camera in a three dimensional space to generate a virtual camera corresponding to the tracked camera.

18. A system for enhancing a broadcast of an event, comprising:

a broadcast server configured to receive audio visual (A/V) data and supplemental data; and

a broadcast client configured to receive the audio visual data and the supplemental data transmitted from the broadcast server over a network, the broadcast client communicating with the broadcast server over the network,

wherein the broadcast client;

generates a synthetic scene based on the audio visual data and the supplemental data;

generates a depth map to store depth information for the synthetic scene, wherein generating the depth map comprises;

establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene;

setting a field of view of the virtual camera to corresponding field of view of the tracked camera;

positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; and

extracting depth information of the synthetic tracked object to generate the depth map; and

refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera; and

integrates the synthetic scene into the broadcast using the depth map.

19. The system of claim 18, further comprising:

a display device coupled to the broadcast client to display the integrated broadcast; and

a viewer control unit coupled to the broadcast client to allow a user to select the synthetic scene to be displayed.

20. The system of claim 18, wherein the broadcast server encodes the audio visual data and the supplemental data to be motion picture expert group compatible data, and wherein the broadcast client decodes the motion picture expert group compatible data to retrieve the audio visual data and the supplemental data.

21. The system of claim 18, wherein the supplemental data comprises:

sensing data from a plurality of sensors placed at the strategic points of a synthetic object at the event;

position and orientation data of the synthetic object from a global positioning system receiver; and

camera tracking data from a tracked camera.

22. A machine-readable medium having executable code to cause a machine to perform a method for enhancing a broadcast of an event, the method comprising:

generating a synthetic scene based on audio visual data and supplemental data received in the broadcast;

generating a depth map to store depth information for the synthetic scene; and

integrating the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises;

establishing the virtual camera tracking data of a tracked camera which defines a viewpoint for the synthetic scene;

setting a field of view of the virtual camera to a corresponding field of view of the tracked camera;

positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object;

extracting depth information of the synthetic tracked object to generate the depth map; and

refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.

23. The machine-readable medium of claim 22, wherein the method further comprises selecting a desired synthetic camera view in response to receiving an input through a user interface.

24. The machine-readable medium of claim 22, wherein the method further comprises:

reconstructing a virtual view of a model, using the position information and camera tracking data; and

extracting the depth information from the virtual view to generate the depth map.

25. A machine-readable medium having executable code to cause a machine to perform a method for enhancing a broadcast of an event, the method comprising:

collecting, at a broadcast server, audio visual data and supplemental data from the event;

transmitting the audio visual data and the supplemented data to a broadcast client over a network;

generating, at the broadcast client, a synthetic scene based on the audio visual data and the supplemental data;

generating a depth map to store depth information for the synthetic scene; and

integrating the synthetic scene into the broadcast using the depth map, wherein the generating the depth map comprises;

establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene;

setting a field of view of the virtual camera to corresponding field of view of the tracked camera;

positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object;

extracting depth information of the synthetic tracked object to generate the depth map; and

refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.

26. The machine-readable medium of claim 25, wherein the method further comprises:

encoding, at the broadcast server, the audio visual data and the supplemental data as motion picture expert group compatible data; and

decoding, at the broadcast client, the motion picture expert group compatible data to retrieve the audio visual data and supplemental data.

27. The machine-readable medium of claim 25, wherein the supplemental data comprises:

sensing data from a plurality of sensors placed at the strategic points of the event;

position and orientation data from a global positioning system receiver; and

camera tracking data from a tracked camera.

28. The machine-readable medium of claim 25, wherein the method further comprises selecting a desired synthetic camera view in response to receiving an input at the broadcast client.