Automatic Event Videoing, Tracking And Content Generation

• US 20070279494A1
• Filed: 04/18/2005
• Published: 12/06/2007
• Est. Priority Date: 04/16/2004
• Status: Active Grant

• Current Assignee: Maxx Holdings, Inc.
• Original Assignee: Maxx Holdings, Inc.
• Inventors: Aman, James, Bennett, Paul
• Granted Patent: US 9,094,615 B2
• Time in Patent Office: Days
• US Class Current: 348/169
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/745 Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2627 : for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80
• CPC Class Codes: H04N 5/268 : Signal distribution or switching
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/278 : Subtitling

Abstract

An automatics system 100 that uses one to three grids 20cm of overhead cameras 20c to first video an event area 2. Overall bandwidth is greatly reduced by intelligent hubs 26 that extract foreground blocks 10m based upon initial and continuously updated background images 2r. The hubs also analyze current images 10c to constantly locate, classify and track in 3D the limited number of expected foreground objects 10. As objects 10 of interest are tracked, the system automatically directs ptz perspective view cameras 40c to follow the activities. These asynchronous cameras 40c limit their images to defined repeatable pt angles and zoom depths. Pre-captured venue backgrounds 2r at each repeatable ptz setting facilitate perspective foreground extraction. The moving background, such as spectators 13, is removed with various techniques including stereoscopic side cameras 40c-b and 40c-c flanking each perspective camera 40c. The tracking data 101 derived from the overhead view 102 establishes event performance measurement and analysis data 701. The analysis results in statistics and descriptive performance tokens 702 translatable via speech synthesis into audible descriptions of the event activities corresponding to overhead 102 and perspective video 202.

Claims

1-98. -98. (canceled)

99. A broadcasting system for producing sporting event broadcasts comprising:

a tracking system for determining the current location of a main game object and for updating a tracking database;

a game clock and official scoring interface system for continuously determining the official game time and score and updating the tracking database;

a game videoing system responsive to the tracking database for continuously capturing video from one or more pan, tilt and/or zoom adjustable cameras and updating a game video database, and

a content assembly and compression system responsive to the tracking database and to the game video database for combining video into a compressed broadcast.

100. The broadcasting system of claim 99 where the tracking system further determines the identity and current location of all participants for updating the tracking database.

101. The broadcasting system of claim 100 where the content assembly and compression system combines information from a performance measurement and analysis database into the compressed broadcast, further comprising:

a performance measurement and analysis system responsive to the tracking database for determining game statistics and performance measurements and updating a performance measurement and analysis database.

102. The broadcasting system of claim 101 where the content assembly and compression system combines a commentator descriptor database and/or its translated synthesized audio into the compressed broadcast, further comprising:

an interface to performance commentators system responsive to the performance measurement and analysis database for generating automated commentator descriptors and updating a commentator descriptor database, where the descriptors are capable of being translated into a synthesized audio stream of game commentary.

103. The broadcasting system of claim 99 where the content assembly and compression system combines at least some of the audio portion of the commentator video/audio database into the compressed broadcast, further comprising:

an interface to performance commentators system for capturing video/audio of game commentators and updating a commentator video/audio database.

104. The broadcasting system of claim 99 where the game videoing system separates the foreground participants and all game objects from the fixed venue and moving fan background for at least some of the captured game video and where the content assembly and compression system includes at least some of the separated foreground participants into the compressed broadcast distinctly from the separated background.

105. The broadcasting system of claim 104 for additionally decoding the event broadcast at which time advertisements are inserted based upon encoded knowledge of the separated foreground and background and optionally upon knowledge of the viewer profile and preferences, further comprising:

a broadcast decoder that may be operated on a remote system that is responsive to the compressed broadcast for reassembling the broadcasted separated foreground onto either a broadcasted separated background and/or a remotely stored background at which time advertisements from a remotely stored ad model database are inserted with or without reference to a remotely stored viewer profile and preference database.

106. A broadcast system for producing sporting event broadcasts, comprising:

An first videoing system for capturing a continuous and contiguous matrix of substantially overhead images and extracting video blocks from these images that wholly contain any contiguous foreground pixels, and for updating an overhead image database with these extracted foreground blocks including an indication of their position within the matrix, and

A content assembly and compression system responsive to the overhead image database for combining the extracted foreground blocks into a compressed broadcast.

107. The broadcast system of claim 106 for additionally decoding the event broadcast, comprising:

a broadcast decoder responsive to the compressed broadcast for reassembling the extracted foreground blocks based upon the indications of their positions within the matrix into a composite overhead view.

108. The broadcast system of claim 106 where the content assembly and compression system is responsive to tracking and game video databases for additionally including game video in the compressed broadcast, further comprising:

a tracking system for determining the current location of a main game object and for updating a tracking database, and

a second videoing system responsive to the tracking database for continuously capturing video from one or more pan, tilt and/or zoom adjustable cameras and updating a game video database.

109. The broadcast system of claim 108 where the tracking system further determines the identity and current location of all participants for updating the tracking database.

110. The broadcast system of claim 109 where the content assembly and compression system additionally includes game clock and official scoring information in the compressed broadcast, further comprising:

a game clock and official scoring interface system for continuously determining the official game time and score and updating the tracking database.

111. The broadcasting system of claim 109 where the content assembly and compression system combines information from a performance measurement and analysis database into the compressed broadcast, further comprising:

a performance measurement and analysis system responsive to the tracking database for determining game statistics and performance measurements and updating a performance measurement and analysis database.

112. The broadcasting system of claim 111 where the content assembly and compression system combines a commentator descriptor database and/or its translated synthesized audio into the compressed broadcast, further comprising:

an interface to performance commentators system responsive to the performance measurement and analysis database for generating automated commentator descriptors and updating a commentator descriptor database, where the descriptors are capable of being translated into a synthesized audio stream of game commentary.

113. The broadcasting system of claim 106 where the content assembly and compression system combines at least some of the audio portion of the commentator video / audio database into the compressed broadcast, further comprising:

an interface to performance commentators system for capturing video/audio of game commentators and updating a commentator video/audio database.

114. The broadcasting system of claim 108 where the game videoing system separates the foreground participants and all game objects from the fixed venue and moving fan background for at least some of the captured game video and where the content assembly and compression system includes at least some of the separated foreground participants into the compressed broadcast distinctly from the separated background.

115. The broadcasting system of claim 108 where the tracking system is responsive to the overhead image database and uses image analysis to track the main game object.

116. The broadcasting system of claim 109 where the tracking system is responsive to the overhead image database and uses image analysis to determine participant identity by recognizing identifying marks placed on the participant'"'"'s uniform and/or equipment.

117. The broadcasting system of claim 109 where the tracking system uses image analysis to determine participant identity by recognizing identifying marks already included on the participant'"'"'s uniform and/or equipment, further comprising:

a player &

referee identification system responsive to the participant'"'"'s current location as stored in the tracking database for capturing images of participants at least including their player numbers and using these images to identify the players and referees, and for subsequently updating the tracking database with the participant identity information.

118. The broadcasting system of claim 109 where the tracking system uses radio frequency triangulation to determine participant identity by locating and decoding signals from transmitters and/or transponders placed on the participant'"'"'s uniform and/or equipment, further comprising:

a transponder tracking system capable of sending to and/or receiving radio frequency or similar signals from transmitters and/or transponders placed on the participant, where the analysis of communicated signals results in a determination of unique identity and current position.

119. A method for producing sporting event broadcasts comprising the steps of:

continuously determining the current location of a main game object and updating a tracking database;

continuously determining the official game time and score and updating the tracking database;

responsive to the tracking database, continuously capturing video from one or more pan, tilt and/or zoom adjustable cameras and updating a game video database, and

responsive to the tracking database and to the game video database, combining video from the game video database into a compressed broadcast.

120. The method of claim 119 further comprising the step of determining the identity and current location of all participants and updating the tracking database.

121. The method of claim 120 further comprising the steps of:

responsive to the tracking database, determining game statistics and performance measurements and updating a performance measurement and analysis database, and

responsive to the performance measurement and analysis database, combining at least some of the information from the performance measurement and analysis database into the compressed broadcast.

122. The method of claim 121 further comprising the steps of:

responsive to the performance measurement and analysis database database, generating automated commentator descriptors and updating a commentator descriptor database, where the descriptors are capable of being translated into a synthesized audio stream of game commentary, and

responsive to the commentator descriptor database, combining at least some of the descriptor database or its audio translation into the compressed broadcast.

123. The method of claim 119 further comprising the steps of:

capturing video/audio of game commentators and updating a commentator video / audio database, and

responsive to the commentator video/audio database, combining at least some of the audio portion of the commentator'"'"'s video/audio stream into the compressed broadcast.

124. The method of claim 119 further comprising the steps of:

separating the foreground participants and all game objects from the fixed venue and moving fan background for at least some of the captured game video, and

responsive to the separated game video, including at least some of the separated foreground participants into the compressed broadcast distinctly from separated background.

125. The method of claim 124 further comprising the steps of:

inserting advertisements not included in the broadcast at the time and place the encoded broadcast is decoded based upon encoded knowledge of the separated foreground and/or background, and

varying the inserted advertisements based upon viewer profile and preference information stored at the time and place of decoding.