Optimizations for live event, real-time, 3D object tracking

• US 20030095186A1
• Filed: 11/20/2001
• Published: 05/22/2003
• Est. Priority Date: 11/20/1998
• Status: Active Grant

• Current Assignee: Maxx Holdings, Inc.
• Original Assignee: Intheplay, Inc.
• Inventors: Aman, James A., Haller, William R.
• Granted Patent: US 7,483,049 B2
• Time in Patent Office: Days
• US Class Current: 348/162
• CPC Class Codes: A63B 2024/0025 Tracking the path or location of one or more users, e.g. players of a game
• CPC Class Codes: A63B 2024/0028 Tracking the path of an object, e.g. a ball inside a soccer pitch
• CPC Class Codes: A63B 2024/0056 for statistical or strategic analysis
• CPC Class Codes: A63B 2102/24 Ice hockey
• CPC Class Codes: A63B 2220/13 Relative positions
• CPC Class Codes: A63B 2220/20 Distances or displacements
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 2225/15 with identification means that can be read by electronic means
• CPC Class Codes: A63B 24/0021 Tracking a path or terminating locations
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/292 Multi-camera tracking

Abstract

Various combinations of camera assembly 500, tracking frequency 510, energy source 520, marker: emission method 530, marker: physical form 540, marker: reflective shape 550, ID: location 560, ID: encoding method 570, ID: obtained 580 and calibration method 590 forming a preferred embodiment 1004 as well as several alternative embodiments for tracking the movement of multiple objects within a predefined area. Camera assembly 500 optionally comprises fixed volume tracking 502, fixed area tracking 504 and movable volume tracking 506. Tracking frequency 510 optionally comprises visible light 512, infrared light 514 and ultraviolet light 516. Energy source 520 optionally comprises ring lights emitting visible or IR frequencies 522, existing lights emitting visible frequencies 524 and existing lights modified to emit non-visible frequencies 526. Marker: emission method 530 optionally comprises retroreflective markers 532, reflective markers 534 and fluorescent markers 536. Marker: physical form 540 optionally comprises spherical 542 and flat 544. Marker: reflective shape optionally comprises uniform circular 552 and non-uniform multi-shape 554. ID: location 560 optionally comprises full body 562 and top surface of body 564. ID: encoding method 570 optionally comprises unique constellation 572 and encoded markings 574. ID: obtained 580 optionally comprises during game surface tracking 582 and outside of game surface tracking 584. Calibration method 590 optionally comprises pre-tracking 592 and simultaneously with tracking 594.

Claims

1. An automated system for tracking the movement of multiple objects within a predefined area comprising:

a fixed area tracking matrix for first detecting the motion of each object in (X, Y) space;

a first algorithm operated on a computer system responsive to the fixed area tracking matrix for determining the (X, Y) location of each object;

a movable volume tracking matrix responsive to the determined (X, Y) locations for controllably detecting the motion of each object in (X, Y, Z) space; and

a second algorithm operated on the computer system responsive to the movable volume tracking matrix for determining the (X, Y, Z) dimensional characteristics of each object and for forming a database representative of each object'"'"'s locations, movements and dimensional characteristics.

2. The system of claim 1 further comprising:

one or more energy sources emitting non-visible energy that is detected by both the area and volume tracking matrices;

flat, visibly transparent markers adhered onto multiple locations on each object that reflect the non-visible energy; and

a third algorithm operated on the computer system responsive to the non-visible energy reflected off the markers for forming a database of related coordinates of each marker on each object.

3. The system of claim 1, wherein the objects are additionally identified, further comprising:

one or more energy sources emitting non-visible energy;

at least one flat, visibly transparent uniquely encoded marker adhered onto the top surface of each object that reflects the non-visible energy for detection by both the area and volume tracking matrices; and

a third algorithm operated on the computer system and responsive to the reflections off the at least one encoded marker detected by the area and volume tracking matrices for including into the database each object'"'"'s identity along with its locations, movements, and dimensional characteristics.

4. The system of claim 1, wherein the objects are first identified and tracked in an outer area that is adjoining the predefined area and subsequently tracked as they enter and move about within the predefined area, further comprising:

one or more energy sources emitting non-visible energy throughout both the outer and predefined areas;

one unique marker or set of markers adhered onto each object that reflects the non-visible energy for detection by both the area and volume tracking matrices;

one or more outer area cameras set up to view the outer area responsive to the non-visible energy;

a third algorithm operated on the computer system responsive to the reflections off the unique marker or set of markers detected by the outer area cameras for forming a first database including the identity of each object as well as the object'"'"'s changing location especially as the object enters the predefined area; and

a fourth algorithm operated on the computer system responsive to the area and volume tracking matrices modified to update the first database of object identity and changing location from the outer area to include additional changing location information based upon each object'"'"'s movements in the predefined area.

5. The system of claim 1 further comprising:

one or more energy sources emitting visible energy that is detected by both the area and volume tracking matrices;

spherical markers adhered onto multiple locations on each object that reflect the visible energy; and

a third algorithm operated on the computer system responsive to the visible energy reflected off the markers for forming a database of related coordinates of each marker on each object.

6. The system of claim 1 further comprising:

one or more energy sources emitting non-visible energy that is detected by both the area and volume tracking matrices;

spherical markers adhered onto multiple locations on each object that reflect the non-visible energy; and

a third algorithm operated on the computer system responsive to the non-visible energy reflected off the markers for forming a database of related coordinates of each marker on each object.

7. An automated system for tracking the movement of multiple objects within a predefined area comprising:

one or more energy sources emitting non-visible energy;

flat, visibly transparent markers adhered onto multiple locations of each object that reflect the non-visible energy;

one or more cameras responsive to the reflected non-visible energy; and

a computer system responsive to the one or more cameras for forming a database of related coordinates of each marker on each object.

8. The system of claim 7 further comprising:

a fixed area tracking matrix for first detecting the motion of each object in (X, Y) space, wherein the computer system is responsive to the fixed area tracking matrix for determining the (X, Y) location of each object; and

a movable volume tracking matrix responsive to the determined (X, Y) locations for controllably detecting the motion of each object in (X, Y, Z) space, wherein the computer system is responsive to the movable volume tracking matrix for forming a database of related coordinates of each marker on each object.

9. The system of claim 7, wherein the objects are additionally identified, further comprising at least one flat, visibly transparent uniquely encoded marker adhered onto the top surface of each object that reflects the non-visible energy, and wherein the computer system is responsive to the reflections off the at least one encoded marker for including into the database each object'"'"'s identity along with the related coordinates of each marker on each object.

10. The system of claim 7, wherein the objects are first identified and tracked in an outer area that is adjoining the predefined area and subsequently tracked as they enter and move about within the predefined area, further comprising:

at least one energy source emitting non-visible energy throughout the outer area;

one unique marker or set of markers adhered onto each object that reflects the non-visible energy; and

one or more outer area cameras set up to view the outer area that detect the non-visible energy, wherein the computer system is responsive to the reflections off the unique marker or set of markers detected by the outer area cameras for forming a second database including the identity of each object as well as the object'"'"'s changing location especially as the object enters the predefined area, and

the computer system updates the second database of object identity and changing location from the outer area to include additional related coordinates of each marker on each object from the predefined area.

11. An automated system for identifying multiple objects within a predefined area comprising:

one or more energy sources emitting non-visible energy;

at least one flat, visibly transparent uniquely encoded marker adhered onto the top surface of each object that reflects the non-visible energy;

one or more cameras responsive to the reflected non-visible energy; and

a computer system responsive to the one or more cameras for forming a database of identities of each object.

12. The system of claim 11 further comprising:

a fixed area tracking matrix for first detecting the motion of each object in (X, Y) space, wherein the computer system is responsive to the fixed area tracking matrix for determining the (X, Y) location of each object; and

a movable volume tracking matrix responsive to the determined (X, Y) locations for controllably detecting the motion of each object in (X, Y, Z) space, wherein the computer system is responsive to the movable volume tracking matrix for determining the (X, Y, Z) dimensional characteristics of each object and for forming a database representative of each object'"'"'s movements and dimensional characteristics and associating these with the existing identities of each object.

13. The system of claim 11, wherein the objects are additionally tracked, further comprising flat, visibly transparent markers adhered onto multiple locations of each object that reflect the non-visible energy, wherein the computer system is responsive to the reflections off the flat markers, forms a database of related coordinates of each marker on each object, and associates these coordinates with the existing identities of each object.

14. The system of claim 11, wherein the objects are first identified and tracked in an outer area that is adjoining the predefined area and subsequently tracked as they enter and move about within the predefined area, further comprising:

one or more energy sources emitting non-visible energy throughout the outer area for reflection off the at least one unique marker; and

one or more outer area cameras set up to view the outer area responsive to the non-visible energy emitted throughout the outer area and reflected off the at least one unique marker, wherein the computer system is responsive to the reflections off the at least one unique marker detected by the outer area cameras for forming a second database including the identity of each object as well as the object'"'"'s changing location especially as the object enters the predefined area, wherein

the computer system updates the second database of object identity and changing location from the outer area to include additional changing location information of each encoded marker from the predefined area.

15. An automated system for first identifying and tracking multiple objects in an outer area that is adjoining a predefined area and subsequently tracking those objects as they enter and move about within the predefined area, the system comprising:

one or more energy sources emitting non-visible energy throughout both the outer and predefined areas;

one unique marker or set of markers adhered onto each object that reflects the non-visible energy;

one or more outer area cameras responsive to the non-visible energy reflected off the markers while the objects are within the outer area;

one or more predefined area cameras responsive to the non-visible energy reflected off the markers while the objects are within the predefined area;

a first algorithm operated on a computer system responsive to the outer area cameras for forming a database including the identity of each object as well as the object'"'"'s changing location especially as the object enters the predefined area; and

a second algorithm operated on the computer system responsive to the predefined area cameras for updating the existing database of object identity and changing location from the outer area to include additional changing location information based upon the object movements in the predefined area.

16. The system of claim 15 further comprising:

a fixed area tracking matrix for first detecting the motion of each object in (X, Y) space, wherein the computer system is responsive to the fixed area tracking matrix for determining the (X, Y) location of each object; and

a movable volume tracking matrix responsive to the determined (X, Y) locations for controllably detecting the motion of each object in (X, Y, Z) space.

17. The system of claim 15, wherein the objects are additionally tracked, further comprising flat, visibly transparent markers adhered onto multiple locations of each object that reflect the non-visible energy, wherein the computer system is responsive to the reflections off the flat markers, forms a database of related coordinates of each marker on each object, and associates these coordinates with the existing identities of each object.

18. The system of claim 15 wherein the one unique marker or set of markers further comprises at least one flat, visibly transparent uniquely encoded marker adhered onto the top surface of each object that reflects the non-visible energy.

19. An automated system for tracking the movement of multiple objects within a predefined area comprising:

one or more energy sources emitting visible energy;

flat markers adhered onto multiple locations of each object that reflect the visible energy;

a multiplicity of cameras responsive to the reflected visible energy; and

a computer system responsive to the multiplicity of cameras for forming a database of related coordinates of each marker on each object.

20. The system of claim 19 wherein the multiplicity of cameras comprises at least two fixed volume tracking cameras with at least partially overlapping fields-of-view for detecting the motion of each marker in (X, Y, Z) space.

21. The system of claim 20 wherein the multiplicity of cameras forms an ever larger total field-of-view by adding additional sets of two or more fixed volume tracking cameras whose individual fields-of-view partially overlap each other covering new volume space and partially overlap existing cameras.

30. The system of claim 19 wherein the multiplicity of cameras forms an ever larger total field-of-view by adding additional sets of two or more fixed volume tracking cameras whose individual fields-of-view partially overlap each other covering new volume space and partially overlap existing cameras.

22. An automated system for tracking the movement of multiple objects within a predefined area comprising:

one or more energy sources emitting non-visible energy;

flat markers adhered onto multiple locations of each object that reflect the non-visible energy;

a multiplicity of cameras responsive to the reflected non-visible energy; and

a computer system responsive to the multiplicity of cameras for forming a database of related coordinates of each marker on each object.

23. The system of claim 22 wherein the multiplicity of cameras comprises at least two fixed volume tracking cameras with at least partially overlapping fields-of-view for detecting the motion of each marker in (X, Y, Z) space.

24. The system of claim 23 wherein the multiplicity of cameras forms an ever larger total field-of-view by adding additional sets of two or more fixed volume tracking cameras whose individual fields-of-view partially overlap each other covering new volume space and partially overlap existing cameras.

25. An automated system for identifying multiple objects within a predefined area comprising:

one or more energy sources emitting visible energy;

at least one uniquely encoded marker adhered onto the top surface of each object that reflects the visible energy;

a multiplicity of cameras responsive to the reflected visible energy; and

a computer system responsive to the multiplicity of cameras for forming a database of identities of each object.

26. The system of claim 25 wherein the multiplicity of cameras comprises at least two fixed volume tracking cameras with at least partially overlapping fields-of-view for detecting the motion of each marker in (X, Y, Z) space.

27. The system of claim 26 wherein the multiplicity of cameras forms an ever larger total field-of-view by adding additional sets of two or more fixed volume tracking cameras whose individual fields-of-view partially overlap each other covering new volume space and partially overlap existing cameras.

28. An automated system for identifying multiple objects within a predefined area comprising:

one or more energy sources emitting non-visible energy;

at least one uniquely encoded marker adhered onto the top surface of each object that also reflects the non-visible energy;

a multiplicity of cameras responsive to the reflected non-visible energy; and

a computer system responsive to the multiplicity of cameras for forming a database of identities of each object.

29. The system of claim 28 wherein the multiplicity of cameras comprises at least two fixed volume tracking cameras with at least partially overlapping fields-of-view for detecting the motion of each marker in (X, Y, Z) space.

31. An automated system for first identifying and tracking multiple objects in an outer area that is adjoining a predefined area and subsequently tracking those objects as they enter and move about within the predefined area, the system comprising:

one or more energy sources emitting visible energy throughout both the outer and predefined areas;

one unique marker or set of markers adhered onto each object that reflects the visible energy;

a multiplicity of outer area cameras responsive to the visible energy reflected off the markers while the objects are within the outer area;

a multiplicity of predefined area cameras responsive to the visible energy reflected off the markers while the objects are within the predefined area;

a first algorithm operated on a computer system responsive to the outer area cameras for forming a database including the identity of each object as well as the object'"'"'s changing location especially as the object enters the predefined area; and

a second algorithm operated on the computer system responsive to the predefined area cameras for updating the existing database of object identity and changing location from the outer area to include additional changing location information based upon the object movements in the predefined area.

32. The system of claim 31 wherein each multiplicity of cameras comprises at least two fixed volume tracking cameras with at least partially overlapping fields-of-view for detecting the motion of each marker in (X, Y, Z) space.

33. The system of claim 32 wherein each multiplicity of cameras forms an ever larger total field-of-view by adding additional sets of two or more fixed volume tracking cameras whose individual fields-of-view partially overlap each other covering new volume space and partially overlap existing cameras.