Method and system for generating a 3D representation of a dynamically changing 3D scene

• US 9,406,131 B2
• Filed: 05/24/2007
• Issued: 08/02/2016
• Est. Priority Date: 06/02/2006
• Status: Active Grant

• Current Assignee: Vizrt AG (Vizrt Group Holding AS)
• Original Assignee: LiberoVision AG (Vizrt Group Holding AS)
• Inventors: Wrmlin, Stephan, Niederberger, Christoph
• Primary Examiner(s): TODD, GREGORY G
• Application Number:
  US12/302,928
• Publication Number:
  US 20090315978A1
• Time in Patent Office: 3,358 Days
• Field of Search: 348/47, 348/48, 348/94, 348/139, 348/157, 348/159, 345/419, 345/679
• US Class Current: 1/1
• CPC Class Codes: A63B 2024/0025 Tracking the path or location of one or more users, e.g. players of a game
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 20/42 of sport video content

Abstract

A method for generating a 3D representation of a dynamically changing 3D scene, which includes the steps of:

• • acquiring at least two synchronised video streams (120) from at least two cameras located at different locations and observing the same 3D scene (102);
  • determining camera parameters, which comprise the orientation and zoom setting, for the at least two cameras (103);
  • tracking the movement of objects (310a,b, 312a,b; 330a,b, 331a,b, 332a,b; 410a,b, 411a,b; 430a,b, 431a,b; 420a,b, 421a,b) in the at least two video streams (104);
  • determining the identity of the objects in the at least two video streams (105);
  • determining the 3D position of the objects by combining the information from the at least two video streams (106);
  • wherein the step of tracking (104) the movement of objects in the at least two video streams uses position information derived from the 3D position of the objects in one or more earlier instants in time.

As a result, the quality, speed and robustness of the 2D tracking in the video streams is improved.

Claims

1. A method for generating a 3D representation of a dynamically changing 3D scene including human players, comprising the steps of:

a) acquiring at least two video streams from at least two cameras located at different locations, observing the same 3D scene, and outputting texture data of said at least two video streams for all frames, wherein each of the at least two cameras are side view cameras which do not look down at a surface of a playing field in a vertical direction;

b) determining camera parameters, which comprise the position, orientation and internal parameters, for said at least two cameras;

c) tracking the 2D movement of players and determining 2D shapes of the players in the at least two video streams based only on the texture data of said at least two video streams, wherein the 2D shapes of players is determined by evaluating the texture data and defining an area that includes at least one of a player'"'"'s position and size;

d) determining the 3D position of the players by combining information from the at least two video streams to generate a 3D computational model of the 3D scene which includes data for rendering a synthesized view of the 3D scene from a virtual viewpoint that is distinct from viewpoints associated with the locations of the at least two cameras; and

e) computing extrapolated 2D positions of the players in the at least two video streams from the 3D positions of the players;

wherein the step of tracking the 2D movement of the players in the at least two video streams and determining the 2D shapes of the players in the at least two video streams uses said computed extrapolated 2D positions of the players in the at least two video streams that stem from one or more earlier instants in time.

2. The method of claim 1, further comprising the step of:

f) determining the position and orientation of 3D rendering objects, each 3D rendering object being part of a computational model, the computational model comprising the 3D representation of the 3D scene, and each 3D rendering object corresponding to a player in the video stream, wherein the 3D rendering objects comprise surfaces on which image information from one or more video streams is rendered when generating the 3D representation of the scene.

3. The method of claim 1, wherein the step of computing extrapolated 2D positions of the players comprises:

computing extrapolated 3D positions of the players from the 3D positions of the players; and

computing the extrapolated 2D positions of the players as projections of the extrapolated 3D positions of the players back into the at least two video streams.

4. The method of claim 1, further comprising the step of determining the identity of players in the at least two video streams using the extrapolated 2D positions of the players that stem from one or more earlier instants in time.

5. The method of claim 1, wherein reference features used for determining camera parameters, are features on the playing field and are identified in a video still image by a user:

pointing, with a graphical input device, to a representation of the same reference feature in a schematic representation of the playing field, and selecting said representation of the reference feature;

pointing, with a graphical input device, to a particular reference feature as seen in the video still image, and selecting said reference feature; and

associating the identity of the representation of the reference feature with the reference feature seen in the still image.

6. The method of claim 5, wherein when selecting said reference feature in the video still image, determining the exact position of the reference feature in the video still image by the steps of:

automatically performing, in the vicinity of the position selected by the user, a feature extraction and in particular an extraction of lines, intersections and corners; and

determining the position of the reference feature as being the position of one of the features extracted, and in particular of a feature whose type is the same as that selected in the schematic representation of the playing field.

7. The method of claim 1, wherein the step of tracking the 2D movement of players and determining 2D shapes of the players in the at least two video streams is implemented by a tracking function and comprises the step of incorporating dynamically changing camera parameters in the tracking function such that the tracking function compensates for changes in the camera parameters during tracking, wherein for each step of locating a particular tracked player in the at least two video streams, at least one of a player'"'"'s expected position and size are corrected according to the known changes of the camera parameters.

8. The method of claim 7, wherein the camera parameters taken into account in the tracking function are camera parameters determined by a camera calibration step performed for the same video frame for which the tracking is done.

9. The method of claim 7, wherein the camera parameters taken into account in the tracking function are camera parameters determined by a camera calibration step performed for one or more previous video frames and are optionally extrapolated.

10. The method of claim 1, wherein the players are categorized as belonging to one of at least two categories, the categories preferably being based on a statistical model and comprising a first team and a second team.

11. The method of claim 1, comprising a segmentation step in which players are separated from the background, comprising the step of:

using alpha channel matching to assign, to each picture element, a value that expresses the probability with which the pixel is part of a player or part of the background.

12. The method of claim 1, comprising a segmentation step in which players are separated from a background, comprising the step of:

after removing the players, filling corresponding holes or unknown parts left in the background by image inpainting and marking such inpainted picture elements as being synthetic image data.

13. The method of claim 12, wherein the segmentation step comprises the step of refining the position and dimensions of bounding boxes around the players.

14. The method of claim 13, wherein the step of determining a background image as seen by the virtual camera further comprises:

rendering predetermined image data on the background model, superimposing it over or replacing the image information provided by the video streams.

15. The method of claim 12, wherein the step of filling holes in the background comprises the step of:

mapping image data that corresponds to source patches comprising real image data to destination patches comprising unknown parts of the image, thereby filling the holes;

wherein the mapping involves a transformation of the patches according to their spatial relationship.

16. The method of claim 12, comprising the step of:

mapping image data that corresponds to source patches comprising real image data to destination patches comprising unknown parts of the image, thereby filling the holes;

wherein the unknown part of the image is known to comprise a landmark feature,

by choosing a destination patch to cover at least part of the landmark feature, and by searching the known image for a matching source patch along the landmark.

17. The method of claim 16, comprising the step of:

for filling a hole comprising a section of a circular landmark, mapping the destination patch to the source patch and vice versa by transforming, preferably rotating scaling, these patches according to their location along the circular landmark.

18. The method of claim 16, comprising the step of:

for filling a hole comprising a section of a straight line landmark, mapping the destination patch to the source patch and vice versa by transforming, preferably scaling, these patches according to their location along the straight line landmark.

19. The method of claim 16, further comprising the step of:

associating a line landmark with a line width;

classifying image elements in source and/or destination patches as being part of the landmark or not, according to said line width;

when searching the known image for a matching source patch, and when copying a source patch, to a destination patch, only considering image elements that are part of the landmark.

20. The method of claim 1, further comprising providing a synthesized view from a virtual viewpoint that is distinct from the camera positions by the steps of:

providing camera parameters of a virtual camera;

determining a background image as seen by the virtual camera on a background model;

determining a projection of each of the players into the virtual camera and superimposing it on the background image; and

outputting the combined image for storage or for further processing.

21. The method of claim 20, wherein the step of determining a background image as seen by the virtual camera comprises the steps of:

blending, for each background picture element, image information from the different video streams that correspond to the same background location;

giving priority to image information that is not marked as being synthetic image data; and

rendering the image information on a background model comprising one or more surfaces representing the background.

22. The method of claim 21, wherein, in the background model, the surface representing the background is a surface representing the playing field or playing field, and optionally also comprises surfaces representing an 3D environment model.

23. The method of claim 20, wherein the step of determining a projection of each of the players into the virtual camera comprises the step of:

rendering the image information from one or more video streams onto 3D rendering objects placed in the 3D background model.

24. A data processing system for generating a 3D representation of a dynamically changing 3D scene, the system comprising:

a computer with a processor, memory, and input output devices, and the data processing system further comprising;

a) a data acquisition module acquiring at least two video streams from at least two cameras located at different locations, observing the same 3D scene, and outputting texture data of said at least two video streams for all frames, wherein each of the at least two cameras are side view cameras which do not look down at a surface of a playing field in a vertical direction;

b) a camera calibration module for determining camera parameters, which comprise the position, orientation and internal parameters, for said at least two cameras;

c) a 2D tracking module that is configured to track the 2D movement of players and determine 2D shapes of the players in the at least two video streams based only on the texture data of said at least two video streams by evaluating the texture data and defining an area that includes at least one of a player'"'"'s position and size;

d) a 3D merging and 3D object position calculation module for determining the 3D position of the players by combining the information determined from the at least two video streams to generate a 3D computational model of the 3D scene which includes data for rendering a synthesized view of the 3D scene from a virtual viewpoint that is distinct from viewpoints associated with the locations of the at least two cameras; and

e) an extrapolation module for computing extrapolated 2D positions of the players in the at least two video streams from the 3D positions of the players;

wherein the 2D tracking module is configured to use extrapolated 2D positions of the players in the at least two video streams that stem from one or more earlier instants in time, and

wherein the computer comprises at least one of;

the data acquisition module, the camera calibration module, the 2D tracking module, the 3D merging and 3D object position calculation module, and the extrapolation module.

25. The data processing system of claim 24, wherein the 3D merging and 3D object position calculation module is configured to provide the function of:

f) determining the position and orientation of 3D rendering objects, each 3D rendering object being part of a computational model, the computational model comprising the 3D representation of the 3D scene, and each 3D rendering object corresponding to a player in the video stream, wherein the 3D rendering objects comprise surfaces that serve to render image information from one or more video streams when generating the 3D representation of the scene.

26. The data processing system of claim 25, comprising a player cutout module for determining:

filled-in background texture data incorporating a flag that specifies whether a particular image patch or pixel is derived from real image data or was generated synthetically,

a player texture and alpha mask for each video stream and each object being tracked, and,

for each player being tracked, a player 2D position and shape and a real-world object identification.

27. The data processing system of claim 26, comprising:

an image synthesis module which provides, from the 3D position of the players, the filled-in background texture data and the player texture and alpha mask video data to a consumer.