System for enhancing a video presentation of a live event

• US 6,597,406 B2
• Filed: 01/26/2001
• Issued: 07/22/2003
• Est. Priority Date: 09/04/1998
• Status: Expired due to Term

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: Gepner, Jerry N., Hsiao, Walter, White, Marvin S., Gloudemans, James R., Cavallaro, Richard H., Honey, Stanley K., O'Brien, Terence J.
• Primary Examiner(s): Harvey; David E.
• Application Number:
  US09/771,238
• Publication Number:
  US 20010005218A1
• Time in Patent Office: 907 Days
• Field of Search: 348/587,586,584,591,705,578,580,581,582,583,585,588,589,590,592,593,596,598
• US Class Current: 348/587
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

A three-dimensional model is created to represent an environment to be captured on video. A camera is fitted with pan, tilt and/or zoom sensors. An operator selects a location in the environment. The three-dimensional model is used to determine the three-dimensional coordinates of the location selected by the operator. Information from the pan, tilt and/or zoom sensors is used to transform the three-dimensional coordinates to a two-dimensional position in the video from the camera. Using the two-dimensional position of the video, a graphic is properly added to the video such that the graphic appears to be at the selected location in the environment.

Claims

1. A method of adding a graphic to a video representation of an environment, said method comprising the steps of:

(a) determining a set of three dimensional coordinates using information identifying a position in said environment and a three dimensional model of at least a portion of said environment, wherein said set of three dimensional coordinates corresponds to said position and includes at least one three dimensional coordinate not listed in said information;

(b) converting said set of three dimensional coordinates into a set of two dimensional positions in said video representation; and

(c) enhancing said video representation with said graphic in an area based on said set of two dimensional positions.

2. The method of claim 1, wherein said step (b) includes the steps of:

(1) receiving view data for said video representation; and

(2) transforming said set of three dimensional coordinates into said set of two dimensional positions in said video representation based at least in part on said view data.

3. The method of claim 1, further including the step of:

(d) receiving said information, wherein said step (d) is performed prior to performing said step (a).

4. The method of claim 1, wherein said position corresponds to a demarcation of a first down.

5. The method of claim 4, wherein said graphic is a first down indicator.

6. The method of claim 1, wherein said information includes one or more vertices of said graphic.

7. The method of claim 6, wherein said graphic is rectangular.

8. The method of claim 1, wherein said environment is a football field, said information identifies a yard line on said football field, and said position includes a portion of said football field extending from said yard line.

9. The method of claim 8, wherein said yard line serves as a line of demarcation for obtaining a first down.

10. The method of claim 1, wherein said graphic is a polygon.

11. The method of claim 1, wherein said step (a) includes the steps of:

(1) setting up a set of points in said three dimensional model in a line corresponding to said position;

(2) determining vertical displacements in said three dimensional model for points in said set of points set up in said step (a)(1); and

(3) setting up a border for said graphic.

12. The method of claim 11, wherein said graphic is a first down indicator.

13. The method of claim 11, wherein said step (a) further includes the step of:

(4) determining whether said line is on a preexisting curve in said three dimensional model,

wherein said step (a)(2) includes the steps of;

(i) if it is determined in said step (a)(4) that there is a preexisting curve, employing an equation for said preexisting curve to determine said vertical displacements, and

(ii) if is determined in said step (a)(4) that there is not a preexisting curve, interpolating between two curves in said three dimensional model to determine said vertical displacements.

14. The method of claim 1, wherein said step (a) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position;

(2) breaking said polygon into a plurality of polygons; and

(3) determining vertical displacements in said three dimensional model for each vertex of each polygon in said set of polygons.

15. The method of claim 1, wherein said step (a) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position; and

(2) determining vertical displacements in said three dimensional model for said corners.

16. The method of claim 1, wherein said graphic is an advertisement.

17. The method of claim 1, wherein said graphic is a first down indicator.

18. The method of claim 1, wherein said step (c) includes the step of:

(1) identifying at least one pixel for exclusion within said area.

19. The method of claim 18, wherein in said step (c) includes the step of:

(2) leaving said at least one pixel unaltered.

20. The method of claim 19, wherein said graphic is a first down indicator.

21. The method of claim 18, wherein said at least one pixel is identified in said step (c)(1) based at least in part on a color of said at least one pixel.

22. The method of claim 1, wherein said step (c) includes the step of:

(1) changing a characteristic of at least one pixel based at least in part on a color of said at least one pixel.

23. The method of claim 1, wherein said information does not list any three dimensional coordinates.

24. The method of claim 1, wherein said at least one three dimensional coordinate not listed corresponds to at least a portion of said position.

25. The method of claim 24, wherein said position corresponds to a first down indicator.

26. The method of claim 1, wherein said information only lists a single dimension coordinate.

27. The method of claim 26, wherein said single dimension coordinate identifies a yard line.

28. The method of claim 1, wherein said information only lists two dimensional coordinates.

29. The method of claim 1, wherein no three dimensional coordinate in said set of three dimensional coordinates is listed in said information.

30. The method of claim 1, wherein said method further includes the step of:

(e) selecting said position, wherein said step (e) is performed before performing said step (a).

31. The method of claim 1, wherein said set of three dimensional coordinates includes two or more three dimensional coordinates.

32. The method of claim 1, wherein said set of three dimensional coordinates includes one or more sets of x, y, and z values.

84. The method of claim 2, wherein said view data identifies pan, tilt, and zoom for a camera providing said video representation.

33. A method of adding a graphic to a video representation of an environment, said method comprising the steps of:

(a) receiving information that identifies a position in said environment for said graphic;

(b) determining a set of three dimensional coordinates using said information and a three dimensional model of at least a portion of said environment, wherein said set of three dimensional coordinates corresponds to said position and includes at least one three dimensional coordinate not listed in said information;

(c) converting said set of three dimensional coordinates into a set of two dimensional positions in said video representation; and

(d) enhancing said video representation with said graphic in an area based on said set of two dimensional positions.

34. The method of claim 33, wherein said position corresponds to a demarcation of a first down.

35. The method of claim 33, wherein said step (c) includes the steps of:

(1) receiving view data for said video representation; and

(2) transforming said set of three dimensional coordinates into said set of two dimensional positions in said video representation based at least in part on said view data.

36. The method of claim 33, wherein said environment is a football field, said information received in said step (a) identifies a yard line on said football field, and said position includes a portion of said football field extending from said yard line.

37. The method of claim 36, wherein said yard line serves as a line of demarcation for obtaining a first down.

38. The method of claim 33, wherein said step (b) includes the steps of:

(1) setting up a set of points in said three dimensional model in a line corresponding to said position;

(2) determining vertical displacements in said three dimensional model for points in said set of points set up in said step (b)(1); and

(3) setting up a border for said graphic.

39. The method of claim 38, wherein said graphic is a first down indicator.

40. The method of claim 38, wherein said step (b) further includes the step of:

(4) determining whether said line is on a preexisting curve in said three dimensional model,

wherein said step (b)(2) includes the steps of;

(i) if it is determined in said step (b)(4) that there is a preexisting curve, employing an equation for said preexisting curve to determine said vertical displacements, and

(ii) if is determined in said step (b)(4) that there is not a preexisting curve, interpolating between two curves in said three dimensional model to determine said vertical displacements.

41. The method of claim 33, wherein said step (b) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position;

(2) breaking said polygon into a plurality of polygons; and

(3) determining vertical displacements in said three dimensional model for each vertex of each polygon in said set of polygons.

42. The method of claim 33, wherein said step (b) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position; and

(2) determining vertical displacements in said three dimensional model for said corners.

43. The method of claim 33, wherein said graphic is an advertisement.

44. The method of claim 33, wherein said graphic is a first down indicator.

45. The method of claim 33, wherein said step (d) includes the step of:

(1) identifying at least one pixel for exclusion within said area.

46. The method of claim 45, wherein in said step (d) includes the step of:

(2) leaving said at least one pixel unaltered.

47. The method of claim 46, wherein said graphic is a first down indicator.

85. The method of claim 34, wherein said graphic is a first down indicator.

48. One or more processor readable storage devices having processor readable code embodied on said one or more processor readable storage devices, said processor readable code for programming one or more processors to perform a method of adding a graphic to a video representation of an environment, said method comprising the steps of:

(a) receiving information that identifies a position in said environment for said graphic;

(b) determining a set of three dimensional coordinates using said information and a three dimensional model of at least a portion of said environment, wherein said set of three dimensional coordinates corresponds to said position and includes at least one three dimensional coordinate not listed in said information;

(c) converting said set of three dimensional coordinates into a set of two dimensional positions in said video representation; and

(d) enhancing said video representation with said graphic in an area based on said set of two dimensional positions.

49. One or more processor readable storage devices according to claim 48, wherein said position corresponds to a demarcation of a first down.

50. One or more processor readable storage devices according to claim 49, wherein said graphic is a first down indicator.

51. One or more processor readable storage devices according to claim 48, wherein said step (c) includes the steps of:

(1) receiving view data for said video representation; and

(2) transforming said set of three dimensional coordinates into said set of two dimensional positions in said video representation based at least in part on said view data.

52. One or more processor readable storage devices according to claim 48, wherein said environment is a football field, said information received in said step (a) identifies a yard line on said football field serving as a line of demarcation for obtaining a first down, and said position includes a portion of said football field extending from said yard line.

53. One or more processor readable storage devices according to claim 48, wherein said step (b) includes the steps of:

(1) setting up a set of points in said three dimensional model in a line corresponding to said position;

(2) determining vertical displacements in said three dimensional model for points in said set of points set up in said step (b)(1); and

(3) setting up a border for said graphic.

54. One or more processor readable storage devices according to claim 53, wherein said graphic is a first down indicator.

55. One or more processor readable storage devices according to claim 53, wherein said step (b) further includes the step of:

(4) determining whether said line is on a preexisting curve in said three dimensional model,

wherein said step (b)(2) includes the steps of;

(i) if it is determined in said step (b)(4) that there is a preexisting curve, employing an equation for said preexisting curve to determine said vertical displacements, and

(ii) if is determined in said step (b)(4) that there is not a preexisting curve, interpolating between two curves in said three dimensional model to determine said vertical displacements.

56. One or more processor readable storage devices according to claim 48, wherein said step (b) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position;

(2) breaking said polygon into a plurality of polygons; and

(3) determining vertical displacements in said three dimensional model for each vertex of each polygon in said set of polygons.

57. One or more processor readable storage devices according to claim 48, wherein said step (b) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position; and

(2) determining vertical displacements in said three dimensional model for said corners.

58. One or more processor readable storage devices according to claim 48, wherein said graphic is an advertisement.

59. One or more processor readable storage devices according to claim 48, wherein said graphic is a first down indicator.

60. One or more processor readable storage devices according to claim 48, wherein said step (d) includes the step of:

(1) identifying at least one pixel for exclusion within said area.

61. One or more processor readable storage devices according to claim 60, wherein in said step (d) includes the step of:

(2) leaving said at least one pixel unaltered.

62. One or more processor readable storage devices according to claim 61, wherein said graphic is a first down indicator.

63. One or more processor readable storage devices according to claim 48, wherein said information does not list any three dimensional coordinates.

64. One or more processor readable storage devices according to claim 48, wherein said at least one three dimensional coordinate not listed corresponds to at least a portion of said position, wherein said position corresponds to a first down indicator.

65. One or more processor readable storage devices according to claim 48, wherein said information only lists a single dimension coordinate, wherein said single dimension coordinate identifies a yard line.

66. One or more processor readable storage devices according to claim 48, wherein said information only lists two dimensional coordinates.

67. One or more processor readable storage devices according to claim 48, wherein no three dimensional coordinate in said set of three dimensional coordinates is listed in said information.

68. One or more processor readable storage devices according to claim 48, wherein said set of three dimensional coordinates includes two or more three dimensional coordinates.

69. One or more processor readable storage devices according to claim 48, wherein said set of three dimensional coordinates includes one or more sets of x, y, and z values.

74. The method of claim 56, wherein said graphic is a first down indicator.

70. An apparatus comprising:

a set of one or more field view sensors adapted to sense field of view information from a camera;

one or more storage devices; and

one or more processors in communication with said one or more storage devices and said set of one or more field view sensors, said one or more processors perform a method of adding a graphic to a video representation of an environment originating from said camera, said method comprising the steps of;

(a) receiving information that identifies a position in said environment for said graphic;

(b) determining a set of three dimensional coordinates using said information and a three dimensional model of at least a portion of said environment, wherein said set of three dimensional coordinates corresponds to said position and includes at least one three dimensional coordinate not listed in said information;

(c) converting said set of three dimensional coordinates into a set of two dimensional positions in said video representation; and

(d) enhancing said video representation with said graphic in an area based on said set of two dimensional positions.

71. The apparatus of claim 70, wherein said position corresponds to a demarcation of a first down.

72. The apparatus of claim 71, wherein said graphic is a first down indicator.

73. The apparatus of claim 70, wherein said step (c) includes the steps of:

(1) receiving said field of view information from said set of one or more field of view sensors for said video representation originating from said camera; and

(2) transforming said set of three dimensional coordinates into said set of two dimensional positions in said video representation based at least in part on said field of view information.

75. The apparatus of claim 70, wherein said step (b) includes the steps of:

(1) setting up a set of points in said three dimensional model in a line corresponding to said position;

(2) determining vertical displacements in said three dimensional model for points in said set of points set up in said step (b)(1); and

(3) setting up a border for said graphic.

76. The apparatus of claim 75, wherein said graphic is a first down indicator.

77. The apparatus of claim 70, wherein said step (b) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position;

(2) breaking said polygon into a plurality of polygons; and

(3) determining vertical displacements in said three dimensional model for each vertex of each polygon in said set of polygons.

78. The apparatus of claim 70, wherein said step (b) includes the steps of:

(1) determining corners in said three dimensional model for a polygon bounding said graphic, wherein said corners correspond to said position; and

(2) determining vertical displacements in said three dimensional model for said corners.

79. The apparatus of claim 70, wherein said graphic is an advertisement.

80. The apparatus of claim 70, wherein said graphic is a first down indicator for a football game.

81. The apparatus of claim 70, wherein said step (d) includes the step of:

(1) identifying at least one pixel for exclusion within said area.

82. The apparatus of claim 81, wherein in said step (d) includes the step of:

(2) leaving said at least one pixel unaltered.

83. The apparatus of claim 82, wherein said graphic is a first down indicator.