Method and apparatus for generating virtual views of sporting events

• US 6,124,862 A
• Filed: 06/12/1998
• Issued: 09/26/2000
• Est. Priority Date: 06/13/1997
• Status: Expired due to Fees

• Current Assignee: Anivision, Inc.
• Original Assignee: Anivision, Inc.
• Inventors: Boyken, Jeffery D., Mitchell, Brian K., Wells, Robert V.
• Primary Examiner(s): Nguyen, Phu K.
• Assistant Examiner(s): Vo, Cliff N.
• Application Number:
  US09/094,524
• Time in Patent Office: 837 Days
• Field of Search: 345/418, 345/419, 345/427, 345/425, 345/428, 345/429, 345/430, 345/433, 345/435
• US Class Current: 345/419
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

A virtual view system uses raw imagery from cameras placed around a sporting arena to generate virtual views of the sporting event from any contemplated view point. The system consists of an optical tracking system, a virtual environment server, and one or more virtual view stations. The optical tracking system receives raw, 2-dimensional video data from a pre-selected number of cameras strategically placed around a sporting arena. The raw 2-dimensional data is then compartmentalized into data gates and processed via a series of software image processors into body state data assigned to each sports player or other targeted entity. The body state data is then passed to a virtual environmental server which generates body position information and visual models for transfer to a selected number of virtual view stations. Each virtual view station includes viewing software for rendering and viewing a virtual sports scene as desired. The view stations also control the viewing point of view of a virtual camera and output video data to a video production center so that video data may be transmitted and combined with other video output as needed. The system permits an operator to select a desired virtual view from the optional control center and transmit the virtual view images to a remote audience. Optional subsystems such as a control center and an archive computer may be integrated into the system to alter camera positioning, tether, focus, and zoom, and to store processed data for the sporting event and replay the data on demand.

Claims

1. An apparatus for generating virtual views of sporting events, comprising:

a. an optical tracking subsection for receiving and converting video data into three dimensional positional data having a camera interface, an I/O sequence subsection, an array image processor subsection having at least one image processor, and a track correlation subsection;

b. a virtual environmental server for receiving said positional data from said optical tracking subsection and generating virtual sports figures data and virtual scene data, said environmental server having a model texturing subsection and at least one environment server computer; and

,

c. a least one virtual view station for rendering said virtual sports figures data and said virtual scene data from said virtual environment server into a virtual image.

2. An apparatus as recited in claim 1, further comprising an archive computer for providing real-time recording and playback of body position information and visual models, said archive computer including means for exchanging data with said optical tracking subsection and said virtual environmental server.

3. An apparatus as recited in claim 1, further comprising a control center for tailoring operation of said apparatus having a controller interface computer and a VVS control center, said control center including means for exchanging data with said virtual view station and said optical tracking subsection.

4. An apparatus as recited in claim 3, wherein said image processor includes means for identifying a three dimensional position of a selected sports player in said video data.

5. An apparatus as recited in claim 4, wherein said optical tracking subsection includes a camera controller for controlling operation of cameras supplying said video data.

6. An apparatus as recited in claim 5, wherein said I/O sequence subsection includes a gate builder for dividing up said video data into suitably sized arrays and an I/O sequencer for sequencing said sized arrays to said array image processor subsection.

7. An apparatus as recited in claim 1, wherein said image processor includes means for segmenting said video data, means for extracting a series of pre-selected features from said video data, and means for classifying said video data according to a set of preselected feature templates.

8. An apparatus as recited in claim 1 further comprising a DMA interface for storing and distributing shared memory data throughout said apparatus.

9. An apparatus as recited in claim 4, wherein said identifying means comprises means for segmenting said video data, means for extracting a series of preselected features from said data, and means for classifying said data according to a set of preselected feature templates.

10. An apparatus for generating virtual views of sporting events, comprising:

a. means for receiving and converting video data into three dimensional positional data;

b. means for receiving said positional data from said receiving and converting means and generating virtual sports figures data and virtual scene data; and

,

c. means for rendering said virtual sports figures data and said virtual scene data from said virtual environment server into a virtual image.

11. An apparatus as recited in claim 10 wherein said means for receiving and converting video data comprises a camera interface, an I/O sequence subsection, an array image processor subsection having at least one image processor, and a track correlation subsection.

12. An apparatus as recited in claim 11 wherein said means for receiving and generating comprises a model texturing subsection for generating three dimension models and textures, and an environmental server computer for providing body position information and visual models to said virtual view station.

13. An apparatus as recited in claim 10 further comprising means for providing real-time recording and playback of said virtual sports figures data and said virtual scene data.

14. An apparatus as recited in claim 10 further comprising means for tailoring operation of said apparatus including a controller interface computer and a VVS control center, said control center including means for exchanging data with said rendering means and said receiving and converting means.

15. A method for generating virtual views of a sporting event, comprising the steps of:

a. receiving video data from cameras positioned around said sporting event;

b. converting said video data into three dimensional positional data of each sporting participant appearing in said video data;

c. generating three dimensional models from said positional data for said participants; and

,

d. combining coherent body position information with said three dimensional models to render a virtual scene.

16. A method as recited in claim 15 wherein said converting step comprises the steps of:

a. partitioning said video data into a plurality of gates;

b. sequencing said gates to a plurality of array processors for image processing; and

,

c. identifying objects in said video data through segmentation, feature extraction, and object classification.

17. A method as recited in claim 16 wherein said generating step comprises the steps of:

a. creating a three dimensional wire-frame representation of each said sporting participant appearing in said video data, said wire-frame including body parts for each said participant; and

,

b. generating model textures for each said body part on said wire-frame representation and mapping said textures onto each said body part.

18. A method as recited in claim 17 including the additional steps of creating coherent body position information from said model textures and said positional data prior to said rendering step.

19. A method as recited in claim 15 including the additional step of controlling said cameras'"'"' position, tether, focus, and zoom to provide video data from a desired location in said sporting event.

20. A method as recited in claim 16 including the additional step of recording said three dimensional position data for playback at a later time.