Optimizations for live event, real-time, 3D object tracking

• US 20090046152A1
• Filed: 10/08/2008
• Published: 02/19/2009
• Est. Priority Date: 11/20/1998
• Status: Active Grant

• Current Assignee: Maxx Holdings, Inc.
• Original Assignee: Maxx Holdings, Inc.
• Inventors: Aman, James A.
• Granted Patent: US 8,965,898 B2
• Time in Patent Office: Days
• US Class Current: 348/157
• CPC Class Codes: A63B 2024/0025 Tracking the path or location of one or more users, e.g. players of a game
• CPC Class Codes: A63B 2102/24 Ice hockey
• CPC Class Codes: A63B 2208/12 specially adapted for children
• CPC Class Codes: A63B 2220/13 Relative positions
• CPC Class Codes: A63B 2220/30 Speed
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 24/0021 Tracking a path or terminating locations
• CPC Class Codes: A63B 71/0605 Decision makers and devices using detection means facilitating arbitration
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/002 Programmed access in sequence to a plurality of record carriers or indexed parts, e.g. tracks, thereof, e.g. for editing; transfer of record carriers from magazine G11B15/68, G11B17/10; G11B17/22
• CPC Class Codes: G11B 27/22 Means responsive to presence or absence of recorded information signals
• CPC Class Codes: Y10S 707/913 Multimedia
• CPC Class Codes: Y10S 707/914 Video
• CPC Class Codes: Y10S 707/915 Image

Abstract

A system 1000, and its alternates 1002 through 1022, for automatically tracking and videoing the movements of multiple participants and objects during an event. System 1000 and its alternates comprise a scalable area tracking matrix of overhead tracking cameras 120c, whose individual field-of-views 120v combine to form a contiguous view 504m of the performance area where the event is being held. As participants 110, and all other necessary objects such as 103 (e.g. a puck in ice-hockey) and 104 (e.g. a stick in ice-hockey) move about during the event, computer 160 analyzes the images from contiguous view 504m to create a real-time tracking database at least including participant and object centroid locations respective to the performance area, and preferably including their identities matched to these ongoing locations. System 1000 and its alternates then employ the real-time database to automatically direct, without operator intervention, one or more side-view cameras such as 140-a, 140-b, 140-c and 140-d, to maintain optimal viewing of the event. The participants and objects may additionally be marked with encoded or non-encoded, visible or non-visible markers, either denoting centroid locations visible from their upper surfaces, and/or non-centroid locations visible from perspective views. The encoded markers are preferably placed on upper surfaces and detectable by computer 160 as it analyzes the images from contiguous overhead field-of-view 504m, thus providing participant and object identities to correspond with ongoing locations, further enhancing algorithms for subsequently controlling side-view cameras 140-a through 140-d. The non-encoded markers are preferably placed on multiple non-centroid locations at least on the participants that are then adjustably viewable by system 1000 and its alternates as the system uses the determined locations of each participant from the overhead view to automatically adjust one or more side-view cameras to tightly follow the participant. The resulting images from side-view cameras 140-a through 140-d may then be subsequently processed to determine the non-encoded marker locations, thus forming a three dimensional model of each participant and objects movements.

Claims

7-1. The method of claim 7 wherein the contiguous field-of-view formed by said first set of stationary overhead cameras is substantially parallel to the ground surface within said entire performance area.

8. The method of claim 8 for automatically videoing the movements of any of the one or more participants and objects as they move about within said predefined performance area, throughout the entire said time duration, further comprising the steps of:

using said determined participant and object identities and locations stored in said real-time tracking database to automatically and individually adjust, without the aid of an operator, the current view of each camera in a second set of multiple stationary cameras, capable of pan, tilt and zoom movement, distinct from said first set of stationary overhead cameras, and

capturing a second stream of video images for viewing and recording from said second set of multiple stationary cameras, wherein said second stream of video images create independent views of one or more of the participants and objects within said entire performance area and wherein said second stream of video images is not used to either determine any participant'"'"'s or object'"'"'s two-dimensional centroid locations or to otherwise update said real-time tracking database.

9. The method of claim 9 for further providing a three dimensional model of the movements of the participants and objects, comprising the additional step of:

analyzing said real-time tracking database and said first and said second streams of video images to determine the ongoing relative three-dimensional coordinates of one or more specific, non-centroid locations on each participant or object and for updating the said real-time tracking database to include the additional three-dimensional coordinates of all detected locations matched with the respective participants and objects.

10. The method of claim 10 further comprising the steps of:

adhering markers onto the one or more locations on each participant and object to be tracked;

detecting the markers using computer based image analysis of said first or said second streams of video in order to determine the ongoing relative three-dimensional coordinates of each marker, and

updating said real-time tracking database to indicate the ongoing relative three-dimensional coordinates of said markers matched with the respective participants and objects.

11. The method of claim 11 wherein said markers are substantially transparent to the participants, further comprising the steps of:

using said markers that reflect, retroreflect or fluoresce primarily non-visible energy and are therefore substantially visibly transparent, and

using one or more energy sources to emit non-visible energy throughout said entire performance area, to be reflected or retroreflected off of said markers, or emitting energy that is fluoresced in the non-visible spectrum by said markers, where the non-visible energy is detectable by said first or said second set of cameras.