System for enhancing the television presentation of an object at a sporting event

• US 5,912,700 A
• Filed: 01/10/1996
• Issued: 06/15/1999
• Est. Priority Date: 01/10/1996
• Status: Expired due to Term

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Fox Sports Productions, Inc. (The Walt Disney Company)
• Inventors: Setos, Andrew G., Cavallaro, Richard H., Honey, Stanley K., Burns, Alan Alexander, Heinzmann, Fred Judson, Hill, David Blyth, Gepner, Jerry Neil, Phillips, Alan C., Olsen, Patrick Wade, Rino, Charles Lawrence, Heidmann, Timothy Paul, Guthart, Harold, Evans, Philip Calvin
• Primary Examiner(s): LIN, BRYAN T
• Application Number:
  US08/585,145
• Time in Patent Office: 1,252 Days
• Field of Search: 348/169, 348/170, 348/171, 348/172, 348/157, 348/61, 348/571, 348/578, 348/584, 348/585, 348/586, 348/589, 348/598, 348/599, 348/600, 473/131, 473/198, 473/199, 473/200, 473/351, 473/353, 473/516, 473/517, 473/569, 473/570
• US Class Current: 348/157
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 2225/30 Maintenance
• CPC Class Codes: A63B 2225/54 Transponders, e.g. RFID
• CPC Class Codes: A63B 43/00 Balls with special arrangements
• CPC Class Codes: A63B 67/14 Curling stone; Shuffleboard; Similar sliding games
• CPC Class Codes: A63B 71/0605 Decision makers and devices using detection means facilitating arbitration
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/088 the inserted signal being digital

Abstract

A system for enhancing the television presentation of an object at a sporting event includes one or more sensors which are used to determine the location of the object. Based on the location of the object and the field of view of a broadcast camera, a processor determines the position of the object in a video frame of the broadcast camera. Once knowing where the object is positioned within the video frame, the television signal can be edited or augmented to enhance the presentation of the object.

Claims

1. A method for enhancing a live video presentation of an object during a sporting event, comprising the steps of:

determining a three dimensional location of said object in real space, said step of determining said three dimensional location being performed during said sporting event;

determining a position of an image of said object in a first live video image using said determined three dimensional location of said object, said step of determining said position being performed during said sporting event; and

enhancing said first live video image based on said position.

2. A method according to claim 1 wherein:

said object is a game object used at a sporting event.

3. A method according to claim 2 wherein:

said game object is a hockey puck.

4. A method according to claim 2 wherein:

said game object is a football.

5. A method according to claim 2 wherein:

said game object is a golf ball.

6. A method according to claim 2 wherein:

said game object is a tennis ball.

7. A method according to claim 2 wherein:

said game object is a baseball.

8. A method according to claim 1, wherein:

said step of determining a three dimensional location uses infrared technology.

9. A method according to claim 1 wherein:

said step of determining a three dimensional location uses radar technology.

10. A method according to claim 1, further including the step of:

delaying said first live video image.

11. A method according to claim 1 further including the step of:

sensing flashes.

12. A method according to claim 1 further including the step of:

sensing pulses from said object.

13. A method according to claim 1 wherein:

said step of determining a three dimensional location includes;

sensing said object, including creating data representing said three dimensional location of said object; and

removing false detections from said data.

14. A method according to claim 13 wherein:

said step of removing false detections includes removing data outside of a defined set of boundaries.

15. A method according to claim 13 wherein:

said step of removing false detections includes subtracting data.

16. A method according to claim 13, wherein:

said step of removing false detections includes temporal subtraction.

17. A method according to claim 13 wherein:

said step of removing false detections includes spectral subraction.

18. A method according to claim 13 wherein:

said step of removing false detections includes filtering data.

19. A method according to claim 13 wherein:

said step of removing false detections includes using a region of play filter.

20. A method according to claim 13 wherein:

said step of removing false detections includes removing data representing exclusion zones.

21. A method according to claim 13 wherein:

said step of removing false detections includes removing data if a flash occurred.

22. A method according to claim 13 wherein:

said step of removing false detections includes removing data below a predefined threshold.

23. A method according to claim 1 further including the step of:

determining a broadcast camera'"'"'s pan, tilt and zoom, said broadcast camera being used to capture said first live video image, said pan, tilt and zoom being used in said step of determining a position.

24. A method according to claim 1 further including the step of:

registering a sensor and a camera, said sensor being used to determine said first three dimensional location, said camera being used to capture said first live video image.

25. A method according to claim 1 wherein:

said step of enhancing said first live video image includes;

creating a second video image; and

combining said second video image with said first live video image.

26. A method according to claim 1 wherein:

said step of enhancing includes adding a graphic to said first live video image.

27. A method according to claim 1 wherein:

said step of enhancing includes adding a graphic to said first live video image at said position.

28. A method according to claim 1 wherein:

said step of enhancing includes adding a graphic to said first live video image logically near said position.

29. A method according to claim 1 wherein:

said step of enhancing includes adding a trajectory to said first live video image.

30. A method according to claim 1 wherein:

said step of enhancing includes adding a trajectory and a target to said first live video image.

31. A method according to claim 1 wherein:

said step of enhancing includes adding audio data.

32. A method according to claim 1 wherein:

said step of enhancing includes adding data to a vertical blanking interval during a broadcast of said sporting event.

33. A method according to claim 1 wherein:

said step of enhancing includes providing a viewer with remote control capabilities.

34. A method according to claim 1 wherein:

said step of enhancing includes creating a video frame and combining said video frame with said first live video image.

35. A method according to claim 1 wherein:

said object is a game object, said game object includes an electromagnetic transmitter adapted to transmit an electromagnetic signal not visible to a human eye.

36. A method according to claim 1 wherein:

said step of determining a three dimensional location includes interpolating said object'"'"'s three dimensional location if said object'"'"'s three dimensional location cannot be determined directly.

37. A method according to claim 1 wherein:

said step of enhancing is finished within one second of said step of determining a three dimensional location.

38. A method according to claim 1 wherein:

said step of enhancing is finished within one third of a second of said step of determining a three dimensional location.

39. A method according to claim 1 wherein:

said step of determining said position includes;

transforming data representing said three dimensional location using a matrix.

40. A method according to claim 1 wherein:

said step of determining a three dimensional location includes determining lines of position.

41. A method according to claim 1 wherein:

said step of determining a three dimensional location includes;

determining lines of position;

grouping pairs of said lines of position;

determining one or more points of closest contact for at least a subset of said pairs of lines of position; and

determining a center of said points of closest contact.

42. A method according to claim 1 wherein:

said step of determining a three dimensional location includes;

determining lines of position;

determining probabilities for spaces near said lines of position; and

finding said location based on said probabilities.

43. A method according to claim 1 further including the step of:

inserting nonvideo data into a television signal, said nonvideo data relating to said position, said television signal including said first live video image.

44. A method according to claim 1 further including the step of:

inserting nonvideo data into a television signal, said data including an address on a network, said television signal including said first live video image.

45. A method according to claim 1 further including the step of:

inserting nonvideo data into a television signal, said data including said first location of said object, said television signal including said first live video image.

46. A method according to claim 1 further including the steps of:

calculating one or more statistics at least partially based on said location of said object.

47. A method according to claim 46 further including the steps of:

inserting said one or more statistics into said first video image.

48. A method according to claim 1 wherein:

said step of enhancing includes adding nonvideo data into a television signal, said television signal includes said first live video image.

49. A method according to claim 48, wherein:

said nonvideo data being used at a remote location to enhance said first video image.

50. A method according to claim 1 wherein said step of determining a three dimensional location includes:

receiving an infrared signal;

creating sensor data representing said reception of said infrared signal; and

removing a false detection from said sensor data, said sensor data being used to determine said three dimensional location of said object.

51. A method according to claim 50 wherein:

said step of removing false detections includes removing sensor data received during a flash.

52. A method according to claim 50 wherein:

said infrared signal is a series of pulses; and

said step of removing false detections includes subtraction.

53. A method according to claim 50 wherein:

the step of determining a three dimensional location of said object further includes the steps of;

determining lines of position;

grouping said lines of position into pairs of lines;

determining one or more closest points of intersection for one or more pairs of lines of position; and

determining a center of said points.

54. A method according to claim 50 further including the step of:

determining field of view data for a camera, said camera being used to capture said first live video image, said step of determining said position uses said field of view data.

55. A method according to claim 50 further including the step of:

creating clusters of said sensor data.

56. A method according to claim 1 further including the step of:

sensing field of view data, said step of determining said position uses said three dimensional location of said object and said field of view data.

57. A method according to claim 1 further including the step of:

capturing said first live video image.

58. A method according to claim 1 further including the steps of:

synchronizing a sensor to said object; and

sensing data using said synchronized sensor, said data being used by said step of determining a three dimensional location of said object.

59. A method according to claim 1 wherein:

said object is mobile.

60. A method for enhancing a live video presentation of an object during a sporting event, comprising the steps of:

sensing said object;

determining said object'"'"'s three dimensional location in real space based on said step of sensing;

capturing a first live video image using a camera, said first live video image includes an image of said object;

determining field of view data for said camera while capturing said first live video image;

determining said object'"'"'s position in said first live video image using said field of view data and said object'"'"'s three dimensional location, said step of determining said object'"'"'s position being performed during said sporting event; and

enhancing said live first video image using said determined position.

61. A method according to claim 60 wherein:

said step of enhancing includes creating a second video image, said second video image including a graphic at a position in said second video image corresponding to said object'"'"'s position in said first live video image.

62. A method according to claim 60 wherein:

said step of enhancing includes creating a second video image, said second video image including a graphic logically near said object'"'"'s position in said first live video image.

63. A method according to claim 60 wherein:

said step of determining said object'"'"'s three dimensional location includes;

determining lines of position;

grouping pairs of said lines of position;

determining one or more points of closest contact for at least a subset of said pairs; and

determining a center of said points of closest contact.

64. A method according to claim 60 wherein:

said step of determining said object'"'"'s three dimensional location includes;

determining lines of position; and

finding said object'"'"'s location using probabilities.

65. A method for enhancing a live television presentation of an object during a sporting event, comprising the steps of:

receiving field of view data for a camera during said sporting event;

receiving sensor data pertaining to said object during said sporting event;

determining a three dimensional location of said object in real space using said sensor data during said sporting event;

determining a position of said object in a video frame of said camera using said field of view data and said three dimensional location, said step of determining a position being performed during said sporting event; and

enhancing said video frame based on said position, said enhanced video frame being part of said live television presentation.

66. A system for enhancing a television presentation of an object during a sporting event, comprising:

means for determining a three dimensional location of said object in real space during said sporting event;

means for capturing a live video image which includes an image of said object, said image of said object being at a particular position in said video image;

means for determining said particular position of the object in said video image during said sporting event using said three dimensional location; and

means for enhancing said live video image at said particular position.

68. A system according to claim 66 further including:

a video modification unit, said video modification receives data from said one or more processors and said camera in order to modify said live video image.

69. A system according to claim 68 further including:

a video delay unit coupled between said broadcast camera and said video modification unit.

70. A system according to claim 66 wherein:

said field of view sensor includes pan and tilt sensors.

71. A system according to claim 66 wherein:

said field of view sensor includes a zoom sensor.

72. A system according to claim 66 further including:

a flash detector in communication with said one or more processors.

67. A system for enhancing a live video presentation of an object during a sporting event, said system to be used in conjunction with a camera, said system comprising:

one or more object sensors;

a field of view sensor, said a field of view sensor senses field of view data for said camera; and

one or more processors in communication with said one or more object sensors and said field of view sensor;

said one or more processors programmed to;

determine a three dimensional location of said object in real space during said sporting event,

determine a position of an image of said object in a live video image from said camera during said sporting event, based on said three dimensional location, and

cause the enhancement of said live video image based on said determined position.

73. A system according to claim 67 wherein:

said one or more object sensors include a radar unit.

74. A system according to claim 67 wherein:

said one or more object sensors include an optical sensor.

75. A system according to claim 67 further including:

a synchronization system in communication with said one or more object sensors.

76. A system according to claim 67 further including:

a video modification unit in communication with said camera and said one or more processors;

a video control unit in communication with said camera;

a replay deck in communication with said video control unit; and

a delay unit in communication with said video control unit and said video modification unit.

77. A system according to claim 67 wherein:

said one or more sensors include a master infrared detector and a slave infrared detector; and

said one or more processors subtract slave infrared detector data from master infrared detector data.

78. A system according to claim 67 wherein:

said one or more sensors include a radar transmitter and a radar receiver.

79. An apparatus according to claim 67 wherein:

said one or more processors further programmed to calculate one or more statistics regarding said object.

80. A system according to claim 67 further including an electromagnetic transmitter, completely disposed inside said object, said electromagnetic transmitter adapted to transmit an electromagnetic signal not visible to a human eye to said one or more object sensors.

81. A system for enhancing a live video presentation of an object at a sporting event, said system to be used in conjunction with a camera, comprising:

a field of view sensor coupled to said camera;

a first infrared pulse detector;

a first infrared sensor unit receiving data from said first infrared pulse detector and from said object; and

one or more processors receiving data from said field of view sensor and said first infrared sensor unit, said processors programmed to determine a three dimensional location of said object in real space during said sporting event, to determine a position of an image of said object in a live video frame from said camera during said sporting event and based on said three dimensional location, and to cause an enhancement of said live video frame based on said determined position.

82. A system according to claim 81 further including:

a flash detector, in communication with said one or more processors.

83. A system according to claim 81 wherein:

said first infrared sensor unit includes a master infrared sensor and a slave infrared sensor.

84. A system according to claim 83 wherein:

said one or more processors includes sensor electronics adapted to remove data outside of a defined boundary and below a defined brightness, said sensor electronics further adapted to subtract slave infrared sensor data from master infrared sensor data.

85. A system according to claim 84 further including:

a second infrared sensor unit.

86. A system according to claim 81 wherein:

said first infrared pulse detector includes;

a photodetector;

an infrared filter in front of said photodetector;

a leading edge threshold detector in communication with said photodetector; and

a trailing edge threshold detector in communication with said photodetector.

87. A system according to claim 81 further including:

a collector box; and

a plurality of infrared pulse detectors connected to said collector box.

88. A system according to claim 87 further including:

at least one flash detector in communication with said collector box.

89. A system according to claim 81 wherein:

said field of view sensor includes a pan sensor, a tilt sensor and a zoom sensor.

90. A system according to claim 81 wherein:

said one or more processors include;

means for performing brightness threshold analysis;

means for performing subtraction; and

means for boundary determinations.

91. A system according to claim 81 wherein:

said one or more processors include;

means for performing field of play filters; and

means for using exclusion zones.

92. A system according to claim 81 further including:

a second infrared sensor unit, said one or more processors receives data from said second infrared sensor unit, said one or more processors uses said data from said first infrared sensor unit and from said second infrared sensor unit to determine said three dimensional location of said object.

93. A system for enhancing a live video presentation of an object at a sporting event, comprising:

a camera, said camera includes a tripod for panning and tilting;

a pan sensor connected to said camera;

a tilt sensor connected to said camera;

a zoom sensor connected to said camera;

a first infrared sensor unit including a first master sensor and a first slave sensor;

a second infrared sensor unit including a second master sensor and a second slave sensor;

a first processor in communication with said first infrared sensor;

a second processor in communication with said second infrared sensor;

a third processor in communication with said first processor, said second processor, said pan sensor, said tilt sensor and said zoom sensor;

a flash detector in communication with said first processor; and

one or more pulse detectors in communication with said first and second infrared sensors.

94. A system according to claim 93 further including:

a memory in communication with said third processor;

a keyer in communication with said third processor;

a video control unit in communication with said camera; and

a delay unit in communication with said video control unit and said keyer.

95. A processor readable storage medium having processor readable code embodied on said processor readable storage medium, said processor readable code for programming one or more processors to perform a method comprising the steps of:

determining a three dimensional location of said object in real space, said step of determining a three dimensional position being performed during said sporting event;

receiving field of view data for a first live video image, said first live video image includes an image of said object, said image of said object being at a particular position in said first live video image;

determining said particular position of said image of said object in said first live video image using said determined location and said field of view data, said step of determining being performed during said sporting event; and

directing an enhancement of said first live video image based on said particular position.