Multiple object tracking system

• US 6,567,116 B1
• Filed: 11/20/1998
• Issued: 05/20/2003
• Est. Priority Date: 11/20/1998
• Status: Expired due to Term

• Current Assignee: Maxx Holdings, Inc.
• Original Assignee: James A. Aman, William R. Haller
• Inventors: Aman, James A., Haller, William R.
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US09/197,219
• Time in Patent Office: 1,642 Days
• Field of Search: H04/N.5225, 348/164, 348/169, 348/170, 348/144, 348/148, 348/157, 348/679, 364/152, 364/175, 382/103, 342/419, 342/457, 342/825.49, 342/61, 702/85, 702/89, 702/92, 702/94, 702/95
• US Class Current: 348/169
• CPC Class Codes: A63B 2024/0025 Tracking the path or location of one or more users, e.g. players of a game
• CPC Class Codes: A63B 2102/24 Ice hockey
• CPC Class Codes: A63B 2208/12 specially adapted for children
• CPC Class Codes: A63B 2220/13 Relative positions
• CPC Class Codes: A63B 2220/30 Speed
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 24/0021 Tracking a path or terminating locations
• CPC Class Codes: A63B 71/0605 Decision makers and devices using detection means facilitating arbitration
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning

Abstract

A system 100 for tracking the movement of multiple objects within a predefined area using a combination of overhead X-Y filming cameras 25 and tracking cameras 24 with attached frequency selective filter 24f. Also employed are perspective Z filming cameras 25 and tracking 24 cameras with filter 24f. Objects to be tracked have been marked to include some form of frequency selective reflective material, such as an ink. Marked objects such as patches 7r and 7l, sticker 9 and tape 4a have been specified and are attachable to said multiple objects. System 100 radiates selected energy 23a throughout the predefined area of tracking that is specifically chosen to reflect off said reflective materials used to mark said objects. Reflected energy such as 7m, 9a and 4b is then received by tracking cameras 24 while all other ambient light is blocked by filter 24f. Local Computer System 60 continuously captures images from said tracking cameras 24 which include only the minimum information created by said reflected energy. System 60 efficiently locates said markings on said multiple objects and uses this location information to analyze synchronously captured unfiltered images from filming cameras 25. Using said location information along with pre-knowledge concerning said multiple objects maximum rate of speed and maximum size as well as calculated information such as the vectors of direction, acceleration and velocity based upon previous movements, system 60 is able to efficiently find and extract the background the portion of the unfiltered images that represents said multiple objects.

Claims

1. An automated system for following the movement of one and more objects within a predefined area using an energy source that emits energy at least including a narrow band of tracking energy, the system comprising:

one and more marks placed upon the one and more objects that reflect only the narrow band of tracking energy;

one and more tracking detectors placed throughout the predefined area that receive the reflected narrow band of tracking energy; and

a computer system responsive to the detectors that analyzes the narrow band of tracking energy received by the detectors and creates a database of movement information related to the one and more marks placed upon the one and more objects.

2. The automated system of claim 1 wherein the narrow band of tracking energy is non-visible.

3. The automated system of claim 2 further comprising one and more lamps emitting the narrow band of non-visible tracking energy.

4. The automated system of claim 3 wherein the one and more lamps are physically separated from the one and more detectors.

5. The automated system of claim 1 wherein the narrow band of tracking energy is constantly radiated, reflected, and received without interruption.

6. The automated system of claim 1 wherein the one and more marks are embedded onto the surface of the one and more objects.

7. The automated system of claim 6 wherein the one and more marks are optionally embedded onto the surface of a material that is attached to the one and more objects in such a way that it remains flat to the surface of the object.

8. The automated system of claim 1 wherein one or more of the marks placed upon each object bears indicia uniquely identifying that object.

9. The automated system of claim 8 wherein the computer system determines the identity of each object based solely upon the information obtained from the one or more indicia bearing marks.

10. The automated system of claim 8 wherein the detectors are one and more first tracking cameras arranged in a grid formation overhead of the predefined area where the one and more objects move, the first tracking cameras being used to track the X-Y movement of the one and more objects.

11. The automated system of claim 10 wherein at least one of the one or more indicia bearing marks are placed upon a top surface of each of the one and more objects such that it is consistently in the view of the overhead first tracking cameras.

12. The automated system of claim 11 wherein the computer system determines the identity of each object based solely upon the information obtained from the at least one indicia bearing mark that was placed upon the top of each one and more object as viewed through the grid of overhead first tracking X-Y cameras.

13. The automated system of claim 1 wherein the detectors are one and more first tracking cameras arranged in a grid formation overhead of the predefined area where the one and more objects move, the first tracking cameras being used to track the X-Y movement of the one and more objects.

14. The automated system of claim 13 wherein additional detectors in the form of one and more second tracking cameras are placed at perspective views to the movement of the one and more objects.

15. The automated system of claim 14 wherein additional detectors in the form of one and more filming cameras are included in order to receive visible light images corresponding to the non-visible light images received by the first and second tracking cameras.

16. The automated system of claim 14 wherein the computer determines Z dimension information about the one and more marks placed upon each of the one and more objects based upon images from the one and more second tracking cameras.

17. The automated system of claim 14 wherein the one and more first and second tracking cameras are further fitted with a lens filter to block visible light.

18. The automated system of claim 15 wherein the computer system receives the visible light images from the filming cameras and stores these images cross-indexed with the tracking information obtained from the first an second tracking cameras.

19. The automated system of claim 18 wherein the location information determined by the computer system using the images of the first and second tracking cameras and pertaining to any and all marks is used to selectively process a portion of the images of the filming cameras by performing a comparison of these images to the pre-known background thereby identifying a foreground object without having to process all the pixels of the images.

20. The automated system of claim 1 wherein the location of each mark is tracked individually.

21. The automated system of claim 20 wherein the shape of the one and more marks provides information about the object'"'"'s orientation.

22. The automated system of claim 21 wherein the spatial arrangement of the one and more marks provides information about the object'"'"'s orientation.

23. The automated system of claim 22 wherein the information calculated by the computer system concerning the location of the one and more marks and therefore of the location of the one and more objects onto which the marks have been placed comprises the current location, direction of movement, velocity, acceleration, and orientation of the object associated with the time the images were captured from which the information was determined.

24. The automated system of claim 9 wherein the information calculated by the computer system is automatically stored in a database cross-linked to and retrievable by the identified object.

25. The automated system of claim 24 wherein the information calculated and automatically stored in the database by the computer system is further analyzed to create new comparison information that is also stored in the database.

26. The automated system of claim 25 further comprising a remote computer system and wherein the information calculated and automatically stored in the database by the computer system as well as the new comparison information is transmittable to the remote computer system.

27. The automated system of claim 26 wherein the remote computer system is capable of recreating a graphic representation of the movement of the tracked objects based upon the transmitted information.

74. The method of claim 13 wherein the step of placing the one and more detectors capable of receiving the non-visible energy throughout the predefined area further comprises the step of arranging a set of first tracking cameras in a grid formation overhead of the predefined area, the first tracking cameras being used to track the X-Y movement of the one and more objects.

75. The method of claim 74 wherein the step of placing the one and more detectors capable of receiving the non-visible energy throughout the predefined area further comprises the step of arranging a set of second tracking cameras that are placed at perspective views to the movement of the one and more objects, the second tracking cameras being used to track the Z movement of the and more objects.

76. The method of claim 75 further comprising the step of placing one and more filming cameras capable of receiving the visible energy throughout the predefined area before the objects enter the predefined area.

77. The method of claim 76 further comprising the step of receiving visible light images from the filming cameras and storing these images cross-indexed with the tracking information obtained from the first and second tracking cameras.

28. An automated system for following the movement of one and more objects within a predefined area using an energy source that emits energy at least including a narrow band of non-visible tracking energy, the system comprising:

one and more flat embedded marks placed upon the one and more objects that reflect only the narrow band of non-visible tracking energy;

one and more first tracking detectors placed throughout the predefined area that receive the narrow band of non-visible tracking energy; and

a computer system responsive to the first tracking detectors that analyzes the narrow band of non-visible tracking energy received by the detectors and creates a database of movement information related to the one and more marks placed upon the one and more objects.

29. The automated system of claim 28 wherein the narrow band of tracking energy is in either the ultraviolet or infrared spectrum.

30. The automated system of claim 28 further comprising one and more lamps emitting the narrow band of non-visible tracking energy.

31. The automated system of claim 30 wherein the one and more lamps are physically separated from the one or more tracking cameras.

32. The automated system of claim 28 wherein the narrow band of tracking energy is constantly radiated, reflected, and received without interruption.

33. The automated system of claim 28 wherein the one and more marks are optionally embedded onto the surface of a material that is attached to the one and more objects in such a way that it remains flat to the surface of the object.

34. The automated system of claim 28 wherein one or more of the marks placed upon each object bears indicia uniquely identifying that object.

35. The automated system of claim 34 wherein the computer system determines the identity of each object based solely upon the information obtained from the one or more indicia bearing marks.

36. The automated system of claim 34 wherein the detectors are one and more first tracking cameras arranged in a grid formation overhead of the predefined area where the one and more objects move, the first tracking cameras being used to track the X-Y movement of the one and more objects.

37. The automated system of claim 36 wherein at least one of the one or more indicia bearing marks are placed upon a top surface of each of the one and more objects such that it is consistently in the view of the overhead first tracking cameras.

38. The automated system of claim 37 wherein the computer system determines the identity of each object based solely upon the information obtained from the at least one indicia bearing mark that was placed upon the top of each one and more object as viewed through the grid of overhead first tracking X-Y cameras.

39. The automated system of claim 38 wherein the detectors are one and more first tracking cameras arranged in a grid formation overhead of the predefined area where the one and more objects move, the first tracking cameras being used to track the X-Y movement of the one and more objects.

40. The automated system of claim 39 wherein additional detectors in the form of one and more second tracking cameras are placed at perspective views to the movement of the one and more objects.

41. The automated system of claim 40 wherein additional detectors in the form of one and more filming cameras are included in order to receive visible light images corresponding to the non-visible light images received by the first and second tracking cameras.

42. The automated system of claim 40 wherein the computer determines Z dimension information about the one and more marks placed upon each of the one and more objects based upon images from the one and more second tracking cameras.

43. The automated system of claim 40 wherein the one and more first and second tracking cameras are further fitted with a lens filter to block visible light.

44. The automated system of claim 41 wherein the computer system receives the visible light images from the filming cameras and stores these images cross-indexed with the tracking information obtained from the first and second tracking cameras.

45. The automated system of claim 44 wherein the location information determined by the computer system using the images of the first and second tracking cameras and pertaining to any and all marks is used to selectively process a portion of the images of the filming cameras by performing a comparison of these images to the pre-known background thereby identifying a foreground object without having to process all the pixels of the images.

46. The automated system of claim 28 wherein the location of each mark is tracked individually.

47. The automated system of claim 46 wherein the shape of the one and more marks provides information about the object'"'"'s orientation.

48. The automated system of claim 47 wherein the spatial arrangement of the one and more marks provides information about the object'"'"'s orientation.

49. The automated system of claim 48 wherein the information calculated by the computer system concerning the location of the one and more marks and therefore of the location of the one and more objects onto which the marks have been placed comprises the current location, direction of movement, velocity, acceleration, and orientation of the object associated with the time the images were captured from which the information was determined.

50. The automated system of claim 28 wherein the information calculated by the computer system is automatically stored in a database cross-linked to and retrievable by the identified object.

51. The automated system of claim 50 wherein the information calculated and automatically stored in the database by the computer system is further analyzed to create new comparison information that is also stored in the database.

52. The automated system of claim 51 further comprising a remote computer system and wherein the information calculated and automatically stored in the database by the computer system as well as the new comparison information is transmittable to the remote computer system.

53. The automated system of claim 52 wherein the remote computer system is capable of recreating a graphic representation of the movement of the tracked objects based upon the transmitted information.

54. An automated system for following the movement of one and more objects within a predefined area using an energy source that emits energy at least including a narrow band of non-visible tracking energy, the system comprising:

one and more flat embedded marks of one and more various shapes placed upon one and more objects that constantly reflect only the narrow band of non-visible tracking energy, wherein at least one of the marks placed on each object bears indicia uniquely identifying that object and is located upon a top surface of the object;

one and more tracking cameras arranged in a grid formation overhead of the predefined area that constantly receive the narrow band of non-visible tracking energy; and

a computer system responsive to the overhead tracking cameras that analyzes the narrow band of non-visible tracking energy received by the cameras and creates a database of movement information related to the one and more marks placed upon the one and more objects.

55. The automated system of claim 54 wherein ultraviolet is used as the non-visible tracking energy.

56. The automated system of claim 54 wherein infrared is used as the tracking energy.

57. The automated system of claim 54 wherein the one and more marks are optionally embedded onto the surface of a material that is attached to the one and more objects in such a way that it remains flat to the surface of the object.

58. The automated system of claim 54 wherein additional one and more perspective tracking cameras are placed at perspective views to the movement of the one and more objects.

59. The automated system of claim 58 wherein additional one and more filming cameras are included in order to receive visible light images corresponding to the images received by the first and second tracking cameras.

60. The automated system of claim 58 wherein the computer determines Z dimension information about the one and more marks placed upon each of the one and more objects based upon images from the one and more perspective tracking cameras.

61. The automated system of claim 60 wherein the computer system receives the visible light images from the filming cameras and stores these images cross-indexed with the tracking information obtained from the overhead and perspective tracking cameras.

62. The automated system of claim 61 wherein the database of movement information includes one or more of the current location, direction of movement, velocity, acceleration, and orientation associated with the time the images were captured and related individually to each of the one and more marks and collectively to each object onto which the marks were placed, the information being cross-linked to and retrievable by each object'"'"'s identity as solely determined by the information obtained from the at least one indicia bearing mark placed upon a top surface of each object.

63. A method for following the movement of one and more objects within a predefined area that contains visible energy for viewing the movement and non-visible energy for tracking the movement, comprising the steps of:

placing one and more marks that only reflect the non-visible tracking energy upon the objects to be tracked before the objects enter the predefined area;

placing one and more detectors capable of receiving the non-visible tracking energy throughout the predefined area before the objects enter the predefined area;

receiving the reflected non-visible tracking energy off the one and more marks with one or more of the one and more tracking detectors placed throughout the predefined area;

analyzing the received reflections of the non-visible tracking energy with a computer attached to the detectors to determine the movement and orientation information of each mark individually; and

determining the movement and orientation of each individual object by associating the determined information relating to each individual mark with that of all other all marks attached to the same object.

64. The method of claim 63 wherein the visible energy and non-visible energy are emitted from energy sources placed throughout the predefined tracking area prior to tracking the objects.

65. The method of claim 64 wherein the energy sources are one and more lamps.

66. The method of claim 63 further comprising the step of first placing a mark upon a material and then attaching one and more of the marked materials to the object to be tracked before the object enters the predefined area.

67. The method of claim 65 wherein the marks placed upon the objects and the materials are flat and embedded onto these surfaces.

68. The method of claim 63 further comprising the step of placing at least one mark bearing a unique indicia that only reflects the non-visible tracking energy upon each object to be tracked before the object enters the predefined area.

69. The method of claim 68 further comprising the step of receiving the reflected non-visible tracking energy off the at least one mark bearing a unique indicia with one or more of the one and more tracking detectors placed throughout the predefined area.

70. The method of claim 69 further comprising the step of analyzing the received reflections of the non-visible tracking energy off the at least one mark bearing the unique indicia with the computer attached to the detectors in order to determine the identity of each individual mark bearing a unique indicia.

71. The method of claim 70 further comprising the step of determining the identity of each individual object based solely upon the determined identity of each individual mark bearing a unique indicia attached to the object and relating the determined identity information to the determined movement and orientation information for the same object.

72. The method of claim 71 further comprising the step of developing statistics relating to individual and select groups of objects by comparing and analyzing the determined movement, orientation and identity information of all individual objects.

73. The method of claim 63 wherein the one and more detectors are tracking cameras capable of receiving the non-visible tracking energy.