Optimizations for live event, real-time, 3D object tracking

• US 7,483,049 B2
• Filed: 11/20/2001
• Issued: 01/27/2009
• Est. Priority Date: 11/20/1998
• Status: Expired due to Fees

• Current Assignee: Maxx Holdings, Inc.
• Original Assignee: Intheplay, Inc.
• Inventors: Aman, James A., Haller, William R.
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US10/006,444
• Publication Number:
  US 20030095186A1
• Time in Patent Office: 2,625 Days
• Field of Search: 348/61, 348/169, 348/157, 348/42, 348/47, 348/143, 348/469, 348/164, 348/162, 348/159, 348/152, 348/153, 348/154, 382/103, 382/104, 382/226, 382/228, 382/224, 382/106, 382/154
• US Class Current: 348/162
• CPC Class Codes: A63B 2024/0025 Tracking the path or location of one or more users, e.g. players of a game
• CPC Class Codes: A63B 2024/0028 Tracking the path of an object, e.g. a ball inside a soccer pitch
• CPC Class Codes: A63B 2024/0056 for statistical or strategic analysis
• CPC Class Codes: A63B 2102/24 Ice hockey
• CPC Class Codes: A63B 2220/13 Relative positions
• CPC Class Codes: A63B 2220/20 Distances or displacements
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 2225/15 with identification means that can be read by electronic means
• CPC Class Codes: A63B 24/0021 Tracking a path or terminating locations
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/292 Multi-camera tracking

Abstract

During a sporting or similar event, participants and objects are automatically tracked using a scalable matrix of fixed, non-adjustable overhead cameras that form a view of the entire performance area. The participants and objects are identified with attached visible or non-visible markers that are detected using image analysis. A real-time database of two-dimensional movement locations is created from the resulting information, which is then used to automatically direct multiple fixed, adjustable side-view cameras for capturing and recording the event. Image analysis of this side-view video detects additional markers providing a three-dimensional model of the participants and objects.

Claims

1. A system for automatically videoing the movements of one or more participants or objects as they move about within the entire performance area of a sporting, theatre or concert event, throughout the entire time duration of the event, comprising:

a first set of multiple stationary overhead cameras, not capable of pan, tilt or zoom movement, sufficient to form a contiguous field-of-view of said entire performance area, for generating a first video stream of images continuously throughout said entire time duration;

a first algorithm operated on a computer system exclusively and solely responsive to said first stream of video images for continuously and simultaneously analyzing the images from all cameras in said first set of multiple stationary cameras in order to detect the presence of each participant or object within each and every camera'"'"'s view, to determine for each detected participant or object its relative two-dimensional centroid coordinates within that view, and to use this determined information from all said first set cameras for updating a real-time tracking database of at least the two-dimensional centroid location of each participant or object, relative to said entire performance area, where said real-time tracking database includes all current and past said centroid locations determined throughout said entire time of said event, and

a second algorithm operated on a computer system exclusively and solely responsive to said real-time tracking database for automatically and individually adjusting the current view of each camera in a second set of multiple stationary cameras, capable of pan, tilt and zoom movement distinct from said first set of stationary cameras, wherein each said second set camera is automatically directed without operator intervention to maintain an independent view of the participants or objects, where said second set of cameras outputs a second stream of video images for viewing and recording, and wherein said second stream of video images is not used to either determine any participant'"'"'s or object'"'"'s two-dimensional centroid locations or to otherwise update said real-time tracking database.

2. The system of claim 1 wherein the contiguous field-of-view formed by said first set of stationary overhead cameras is substantially parallel to the ground surface within said entire performance area.

3. The system of claim 1 for further providing a three dimensional model of the movements of one or more participants or objects, further comprising:

a third algorithm operated on a computer system responsive to said real-time tracking database and said first and said second streams of video images for determining the ongoing relative three-dimensional coordinates of one or more specific, non-centroid locations on each participant or object and for updating said real-time tracking database to include the additional three-dimensional coordinates of all detected locations matched with the respective participants or objects.

4. The system of claim 3 further comprising:

markers adhered onto one or more locations on each participant or object to be tracked, where said markers are detectable by said first or said second set of stationary cameras, and where said first or said third algorithms now determine the locations of said markers and update said real-time tracking database with the related ongoing three-dimensional coordinates of each marker for forming the three dimensional model of each participant'"'"'s movements.

5. The system of claim 4 wherein said markers are substantially transparent to the participants, further comprising:

said markers adhered onto participants that reflect, retroreflect or fluoresce primarily non-visible energy and are therefore substantially visibly transparent, and

one or more energy sources emitting non-visible energy that is reflected or retroreflected off of said markers, or emitting energy that is fluoresced in the non-visible spectrum by said markers and is detectable by said first or said second set of stationary cameras.

6. The system of claim 5 wherein the participants are additionally identified, further comprising:

at least one uniquely encoded marker adhered onto an upper surface of each participant to be identified, and

a forth algorithm operated on a computer system for locating and recognizing said encoded markers within said first stream of video images and for updating said real-time tracking database with each participant'"'"'s identity matched to their coordinates.

7. The system of claim 6 wherein said uniquely encoded markers are substantially transparent to the participants, further comprising:

said uniquely encoded markers that reflect, retroreflect or fluoresce primarily non- visible energy and are therefore substantially visibly transparent, and

one or more energy sources emitting non-visible energy that is reflected or retroreflected off of said encoded markers, or emitting energy that is fluoresced in the non-visible spectrum by said encoded markers and is detectable by said first set of stationary overhead cameras.

8. The system of claim 1 wherein the participants are additionally identified, further comprising:

at least one uniquely encoded marker adhered onto an upper surface of each participant to be identified, and

a third algorithm operated on a computer system for locating and recognizing said encoded markers within said first stream of video images and for updating said real-time tracking database with each participant'"'"'s identity matched to their coordinates.

9. The system of claim 8 wherein said uniquely encoded markers are substantially transparent to the participants, further comprising:

said uniquely encoded markers that reflect, retroreflect or fluoresce primarily non-visible energy and are therefore substantially visibly transparent, and

one or more energy sources emitting non-visible energy that is reflected or retroreflected off of said markers, or emitting energy that is fluoresced in the non-visible spectrum by said markers and is detectable by said first set of stationary overhead cameras.

10. The system of claim 1 for videoing the movements of two or more participants, wherein said second set of stationary cameras, capable of pan, tilt and zoom movement, comprises at least two cameras and wherein said second set is additionally directed to automatically reassign any one or more of said second set cameras currently following any one or more participants to instead follow a different one or more participants based upon which camera views are currently blocked by one or more participants in front of another, or upon which camera views are best situated to capture the programmatically desired view of any given participant or game object.

11. A method for automatically videoing the movements of one or more participants or objects as they move about within the entire performance area of a sporting, theatre or concert event, throughout the entire time duration of the event, comprising the steps of:

capturing a first stream of video images using a first set of multiple stationary overhead cameras, not capable of pan, tilt or zoom movement, sufficient to form a contiguous field-of-view of said entire performance area, and where each said first set camera provides images continuously throughout said entire time duration;

simultaneously analyzing said first stream of video images from said first set of multiple stationary overhead cameras in order to continuously detect the presence of any one or more participants or objects within each and every camera'"'"'s view, to determine each detected participant'"'"'s or object'"'"'s relative two-dimensional centroid coordinates within that view, and to use this determined information from all said first set cameras for undatine a real-time tracking database of the two-dimensional centroid location of each participant or object, relative to said entire performance area where said real-time tracking database includes all current and past said centroid locations determined throughout said entire time of said event;

using said determined participant and object locations stored in said real-time tracking database to automatically and individually adjust, without the aid of an operator, the current view of each camera in a second set of multiple stationary cameras, capable of pan, tilt and zoom movement, distinct from said first set of stationary overhead cameras, and

capturing a second stream of video images for viewing and recording from said second set of multiple stationary cameras, wherein said second stream of video images create independent views of one or more of the participants or objects within said entire performance area and wherein said second stream of video images is not used to either determine any participant'"'"'s or object'"'"'s two-dimensional centroid locations or to otherwise update said real-time tracking database.

12. The method of claim 11 wherein the contiguous field-of-view formed by said first set of stationary overhead cameras is substantially parallel to the ground surface within said entire performance area.

13. The method of claim 11 for further providing a three dimensional model of the movements of the participants or objects, comprising the additional step of:

analyzing said real-time tracking database and said first and said second streams of video images to determine the ongoing relative three-dimensional coordinates of one or more specific, non-centroid locations on each participant or object and for updating said real-time tracking database to include the additional three-dimensional coordinates of all detected locations matched with the respective participants or objects.

14. The method of claim 13 further comprising the steps of:

adhering markers onto the one or more locations on each participant or object to be tracked;

detecting said markers using computer based image analysis of said first or said second streams of video in order to determine the ongoing relative three-dimensional coordinates of each marker, and

updating said real-time tracking database to indicate the ongoing relative three-dimensional coordinates of said markers matched with the respective participants or objects.

15. The method of claim 14 wherein said markers are substantially transparent to the participants, further comprising the steps of:

using said markers that reflect, retroreflect or fluoresce primarily non-visible energy and are therefore substantially visibly transparent, and

using one or more energy sources to emit non-visible energy throughout said entire performance area, to be reflected or retroreflected off of said markers, or emitting energy that is fluoresced in the non-visible spectrum by said markers, where the non-visible energy is detectable by d first or said second set of stationary cameras.

16. The method of claim 15 wherein the participants are additionally identified, further comprising the steps of:

placing at least one uniquely encoded marker onto an upper surface of each participant to be identified;

detecting and recognizing each encoded marker using computer based image analysis of said first stream of video images, and

updating said real-time tracking database to indicate the identity of each participant matched to their coordinates.

17. The method of claim 16 wherein the uniquely encoded markers are substantially transparent to the participants, further comprising the steps of:

using said uniquely encoded markers that reflect, retroreflect or fluoresce primarily non-visible energy and are therefore substantially visibly transparent, and

using one or more energy sources to emit non-visible energy throughout said entire performance area, to be reflected or retroreflected off of said encoded markers, or emitting energy that is fluoresced in the non-visible spectrum by said encoded markers, where the non-visible energy is detectable by said first set of stationary overhead cameras.

18. The method of claim 11 wherein the participants are additionally identified, further comprising the steps of:

placing at least one uniquely encoded marker onto an upper surface of each participant to be identified;

detecting and recognizing each said encoded marker using computer based image analysis of said first stream of video images, and

updating said real-time tracking database to indicate the identity of each participant matched to their coordinates.

19. The method of claim 18 wherein uniquely encoded markers are substantially transparent to the participants, further comprising the steps of:

using uniquely encoded markers that reflect, retroreflect or fluoresce primarily non-visible energy and are therefore substantially visibly transparent, and

using one or more energy sources to emit non-visible energy throughout said entire performance area, to be reflected or retroreflected off of said encoded markers, or emitting energy that is fluoresced in the non-visible spectrum by said encoded markers, where the non-visible energy is detectable by said first set of stationary overhead cameras.

20. The method of claim 11 for videoing the movements of two or more participants from optimal viewpoints, further comprising the steps of:

using two or morn cameras in said second set of stationary cameras, and

during the step of individually adjusting the pan, tilt or zoom movements of each camera in said second set of stationary cameras, capable of pan, tilt and zoom movement, dynamically considering the location of each participant or object with respect to the view of each said second set camera and automatically reassigning any one or more said second set cameras currently following any one or more participants or objects to instead follow a different one or more participants or objects based upon which camera views are best situated to capture the programmatically desired view of any given participant or game object.