COORDINATION AND COMBINATION OF VIDEO SEQUENCES WITH SPATIAL AND TEMPORAL NORMALIZATION

• US 20090324191A1
• Filed: 09/09/2009
• Published: 12/31/2009
• Est. Priority Date: 11/24/1999
• Status: Active Grant

• Current Assignee: Dartfish
• Original Assignee: Dartfish
• Inventors: Vetterli, Martin F., Bergonzoli, Victor A., Ayer, Serge J., Reusens, Emmanuel
• Granted Patent: US 8,675,021 B2
• Time in Patent Office: Days
• US Class Current: 386/353
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Given two video sequences, a composite video sequence can be generated (15) which includes visual elements from each of the given sequences, suitably synchronized (11) and represented in a chosen focal plane. A composite video sequence can be made also by similarly combining a video sequence with an audio sequence. In the composite video sequence, contestants, action figures or objects can be shown against a common background (12) even if the given video sequences differ as to background, with the common background taken from one or the other of the given sequences, for example. Alternatively, a different suitable background can be used, e.g. as derived from the given video sequences, as obtained from another video sequence or image, or as otherwise synthesized.

Claims

1-58. -58. (canceled)

59. A computer method for generating a composite video sequence from a plurality of given video sequences for display on a display device, wherein each of the given video sequences comprises a representation of a respective one of a plurality of objects with action relative to a respective environment, the method comprising the steps of:

(a) forming a background representation, said background representation including a common environment;

(b) aligning said video sequence, based on a matching in size and position of elements from the background and/or from the foreground which are present in two video sequences;

(c) forming the composite video sequence based on the formed background; and

(d) displaying the composite video sequence on the display device.

60. The method of claim 59, wherein step (a) comprises forming the background representation independent of any of the objects.

61. The method of claim 60, wherein the background representation is a single image recorded by a camera.

62. The method of claim 60, wherein forming the background representation comprises panoramically combining images from an image sequence.

63. The method of claim 62, wherein the image sequence is one of the given video sequences.

64. The method of claim 63, comprising updating the background representation so as to give preferential weight to more recent images of the given video sequence.

65. The method of claim 60, wherein the background representation comprises a parametric environmental representation.

66. The method of claim 60, wherein forming the background representation comprises associating a reliability measure to representational elements.

67. The method of claim 59, further comprising estimating a geometrical relation between a given video frame and a corresponding background representation.

68. The method of claim 67, wherein estimating comprises image analysis.

69. The method of claim 67, wherein estimating comprises taking into account instrument data furnished by an instrumented camera used in generating one of the given video sequences.

70. The method of claim 67, comprising (i) using the geometrical relation for aligning a given video frame and a corresponding background representation and (ii) associating with pixels a discrepancy measure between the aligned video frame and the background representation.

71. The method of claim 70, comprising using the discrepancy measure in determining a probability for a pixel of a given video frame to belong to background.

72. The method of claim 71, comprising using the discrepancy measure in classifying a pixel of a given video frame as belonging either to background or else to a foreground object.

73. The method of claim 72, wherein classifying takes auxiliary information into account.

74. The method of claim 72, comprising using the classification in inferring statistical data for the foreground object.

75. The method of claim 72, comprising using the classification in replacing the background with a different background.

76. The method of claim 59, wherein each of the given video sequences comprises representations of a respective plurality of objects with action relative to the respective environment.

77. The method of claim 59, wherein the elements are canonical elements.

78. The method of claim 59, comprising automated selection of the elements.

79. The method of claim 59, comprising accepting input for the selection of the elements.

80. The method of claim 59, further comprising identifying time of execution at least at one control point in the execution of an action performance.

81. The method of claim 80, wherein the identification is based at least in part on visual information in one of the given video sequence.

82. The method of claim 80, wherein the identification is based at least in part on an auxiliary signal.

83. The method of claim 82, wherein the auxiliary signal is one of an audio signal, an ultrasound signal, a microwave signal, a radar signal and an optical detector signal.

84. The method of claim 80, comprising storing identified time for later use along with a video sequence.

85. The method of claim 80, wherein time identification is automated.

86. The method of claim 80, comprising accepting input for time identification.

87. The method of claim 80, comprising determining a temporal transformation from one set of control points onto another.

88. The method of claim 80, comprising applying a temporal transformation to at least one of the given video sequences to generate a transformed sequence whose pace is adjusted for temporal matching at control points of given video sequences.

89. The method of claim 88, wherein one of the given video sequences is a reference video sequence, and wherein the temporal transformation is applied only to another given video sequence.

90. The method of claim 88, wherein the temporal transformation comprises offsetting an action performance in one given video sequences with an action performance in another given video sequence.

91. The method of claim 90, wherein offsetting is for controlling playback of a recorded given video so that time of execution at control points corresponds to time of execution in a live-performance given video.

92. The method of claim 88, comprising tagging/indexing/marking video frames of a given video sequence which represent a state of action at the control points.

93. The method of claim 80, comprising applying a nonlinear temporal transformation to at least one of the video sequences.

94. The method of claim 93, wherein the transformation minimizes a global error measure between the video sequences.

95. The method of claim 94, comprising searching for minimization.

96. The method of claim 95, wherein searching is based on a dynamic programming technique.

97. The method of claim 59, further comprising choosing the respective environment of one of the given video sequences as the common environment.

98. The method of claim 59, further comprising_choosing the common environment to be different from the respective environments of each of the given video sequences.

99. The method of claim 59, wherein step (c) comprises blending in forming the composite video sequence.

100. The method of claim 59, wherein step (c) comprises split-screen arranging in forming the composite video sequence.

101. The method of claim 59, wherein the given video sequences are from two sports performances.

102. The method of claim 101, wherein the two sports performances are in different respective environments.

103. The method of claim 102, wherein one of the sports performances is a reference performance.

104. The method of claim 103, wherein the reference performance is one of a world record performance, a national record performance, and a best personal performance.

105. The method of claim 103, wherein the performance other than the reference performance is one of a recent performance and a live performance.

106. The method of claim 101, wherein the sport is one of long jump, triple jump, high jump, pole vault, 100-meter race, and golf swing.

107. The method of claim 77, wherein the canonical elements are one of long-jump running pit lines, long-jump take-off board, 100-m race lanes, pole-vault poles, pole-vault bars, and high-jump bars.

108. A system for generating a composite video sequence from a plurality of given video sequences wherein each of the given video sequences comprises a representation of a respective one of a plurality of objects with action relative to a respective environment, the system comprising:

(a) means for forming a background representation, said background representation including a common environment;

(b) means for aligning said video sequence, based on a matching in size and position of elements from the background and/or from the foreground which are present in two video sequences with a visual element in another of the given video sequences, wherein said visual elements are background elements; and

(c) means for forming the composite video sequence based on the formed background representation; and

(d) a display device for displaying the composite video sequence.

109. A tangible medium comprising a representation of computer instructions for generating a composite video sequence from a plurality of given video sequences and displaying the composite video sequence on a display device, wherein each of the given video sequences comprises a representation of a respective one of a plurality of objects with action relative to a respective environment, the computer instructions comprising:

(a) instructions for forming a background representation, said background representation including a common environment;

(b) instructions for aligning said video sequence, based on a for matching in size and position of elements from the background and/or from the foreground which are present in two video sequences;

(c) instructions for forming the composite video sequence based on the formed background; and

(d) instructions for displaying the composite video sequence on the display device.

110. A computer method for generating a composite video sequence from a plurality of given video sequences and displaying the composite video sequence on a display device, wherein each of the given video sequences comprises a representation of a respective one of a plurality of objects with action relative to a respective environment, the method comprising the steps of:

(a) forming a background representation, said background representation including a common environment, wherein for each video sequence a weighted mask sequence is computed, said weighted mask sequence describing the probability of a pixel to belong to the background;

(c) aligning said video sequences, based on a matching in size and position of elements from the background and/or from the foreground which are present in two video sequences;

(d) forming the composite video sequence based on the formed background representation; and

(e) displaying the composite video sequence on the display device.