Coordination and combination of video sequences with spatial and temporal normalization

• US 8,675,021 B2
• Filed: 09/09/2009
• Issued: 03/18/2014
• Est. Priority Date: 11/24/1999
• Status: Active Grant

• Current Assignee: Dartfish
• Original Assignee: Dartfish
• Inventors: Reusens, Emmanuel, Vetterli, Martin, Ayer, Serge, Bergonzoli, Victor
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US12/555,841
• Publication Number:
  US 20090324191A1
• Time in Patent Office: 1,651 Days
• Field of Search: 345/629, 345/634, 345/639, 345/640, 348/584, 348/586
• US Class Current: 345/639
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Given two video sequences, a composite video sequence can be generated (15) which includes visual elements from each of the given sequences, suitably synchronized (11) and represented in a chosen focal plane. A composite video sequence can be made also by similarly combining a video sequence with an audio sequence. In the composite video sequence, contestants, action figures or objects can be shown against a common background (12) even if the given video sequences differ as to background, with the common background taken from one or the other of the given sequences, for example. Alternatively, a different suitable background can be used, e.g. as derived from the given video sequences, as obtained from another video sequence or image, or as otherwise synthesized.

Claims

1. A computer method for generating a composite video sequence from a plurality of given video sequences for display on a display device in communication with a computer processor, wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the method comprising the steps of:

(a) forming a background representation, using the computer processor, said background representation including a common environment;

(b) aligning said plurality of given video sequences, using the computer processor, based on a matching in size and position of elements, from a background and/or from a foreground, which are present in at least two of the plurality of given video sequences;

(c) forming the composite video sequence, using the computer processor, based on the formed background representation and the aligned video sequences; and

(d) displaying the composite video sequence on the display device.

2. The method of claim 1, wherein step (a) comprises forming the background representation independent of any of the plurality of elements.

3. The method of claim 2, wherein the background representation is a single image recorded by a camera.

4. The method of claim 3, wherein forming the background representation comprises panoramically combining images from an image sequence.

5. The method of claim 4, wherein the image sequence is one of the given video sequences.

6. The method of claim 5, comprising updating the background representation so as to give preferential weight to more recent images of the given video sequence.

7. The method of claim 2, wherein the background representation comprises a parametric environmental representation.

8. The method of claim 2, wherein forming the background representation comprises associating a reliability measure to representational elements.

9. The method of claim 1, further comprising estimating a geometrical relation between a given video frame and a corresponding background representation.

10. The method of claim 9, wherein estimating comprises image analysis.

11. The method of claim 9, wherein estimating comprises taking into account instrument data furnished by an instrumented camera used in generating one of the given video sequences.

12. The method of claim 9, comprising (i) using the geometrical relation for aligning a given video frame and a corresponding background representation and (ii) associating with pixels a discrepancy measure between the aligned video frame and the background representation.

13. The method of claim 12, comprising using the discrepancy measure in determining a probability for a pixel of a given video frame to belong to background.

14. The method of claim 13, comprising using the discrepancy measure in classifying a pixel of a given video frame as belonging either to background or else to a foreground element.

15. The method of claim 14, wherein classifying takes auxiliary information into account.

16. The method of claim 14, comprising using the classification in inferring statistical data for the foreground element.

17. The method of claim 14, comprising using the classification in replacing the background with a different background.

18. The method of claim 1, wherein each of the given video sequences comprises representations of a respective plurality of elements with action relative to the respective environment.

19. The method of claim 1, wherein the elements are canonical elements.

20. The method of claim 1, comprising automated selection of the elements.

21. The method of claim 1, comprising accepting input for the selection of the elements.

22. The method of claim 1, further comprising identifying time of execution at least at one control point in the execution of an action performance.

23. The method of claim 22, wherein the identification is based at least in part on visual information in one of the given video sequence.

24. The method of claim 22, wherein the identification is based at least in part on an auxiliary signal.

25. The method of claim 24, wherein the auxiliary signal is one of an audio signal, an ultrasound signal, a microwave signal, a radar signal and an optical detector signal.

26. The method of claim 22, comprising storing identified time for later use along with a video sequence.

27. The method of claim 22, wherein time identification is automated.

28. The method of claim 22, comprising accepting input for time identification.

29. The method of claim 22, comprising determining a temporal transformation from one set of control points onto another.

30. The method of claim 22, comprising applying a temporal transformation to at least one of the given video sequences to generate a transformed sequence whose pace is adjusted for temporal matching at control points of given video sequences.

31. The method of claim 30, wherein one of the given video sequences is a reference video sequence, and wherein the temporal transformation is applied only to another given video sequence.

32. The method of claim 30, wherein the temporal transformation comprises offsetting an action performance in one given video sequences with an action performance in another given video sequence.

33. The method of claim 32, wherein offsetting is for controlling playback of a recorded given video so that time of execution at control points corresponds to time of execution in a live-performance given video.

34. The method of claim 30, comprising tagging/indexing/marking video frames of a given video sequence which represent a state of action at the control points.

35. The method of claim 22, comprising applying a nonlinear temporal transformation to at least one of the video sequences.

36. The method of claim 35, wherein the transformation minimizes a global error measure between the video sequences.

37. The method of claim 36, comprising searching for minimization.

38. The method of claim 37, wherein searching is based on a dynamic programming technique.

39. The method of claim 1, further comprising choosing the respective environment of one of the given video sequences as the common environment.

40. The method of claim 1, further comprising choosing the common environment to be different from the respective environments of each of the given video sequences.

41. The method of claim 1, wherein step (c) comprises blending in forming the composite video sequence.

42. The method of claim 1, wherein step (c) comprises split-screen arranging in forming the composite video sequence.

43. The method of claim 1, wherein the given video sequences are from two sports performances.

44. The method of claim 43, wherein the two sports performances are in different respective environments.

45. The method of claim 44, wherein one of the sports performances is a reference performance.

46. The method of claim 45, wherein the reference performance is one of a world record performance, a national record performance, and a best personal performance.

47. The method of claim 45, wherein the performance other than the reference performance is one of a recent performance and a live performance.

48. The method of claim 43, wherein the sport is one of long jump, triple jump, high jump, pole vault, 100-meter race, and golf swing.

49. The method of claim 19, wherein the canonical elements are one of long-jump running pit lines, long-jump take-off board, 100-m race lanes, pole-vault poles, pole-vault bars, and high-jump bars.

50. The method of claim 1, further comprising the step of recording by an image recording device the plurality of given video sequences.

51. A system for generating a composite video sequence from a plurality of given video sequences wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the system comprising:

(a) a computer processor forming a background representation, said background representation including a common environment;

(b) said computer processor aligning said plurality of video sequences, based on a matching in size and position of elements from a background and/or from a foreground which are present in at least two of the plurality of given video sequences with a visual element in another of the given video sequences, wherein said visual elements are background elements;

(c) said computer processor further forming the composite video sequence based on the formed background representation and the aligned video sequences; and

(d) a display device in communication with the computer processor, said display device displaying the composite video sequence.

52. A non-transitory tangible medium comprising a representation of computer instructions for generating a non-transitory composite video sequence from a plurality of given video sequences and displaying the composite video sequence on a display device, wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the computer instructions comprising:

(a) instructions for forming a background representation, said background representation including a common environment;

(b) instructions for aligning said plurality of video sequences, based on a matching in size and position of elements from a background and/or from a foreground which are present in at least two of given video sequences;

(c) instructions for forming the non-transitory composite video sequence based on the formed background representation and the aligned video sequences; and

(d) instructions for displaying the non-transitory composite video sequence on the display device.

53. A computer method for generating a composite video sequence from a plurality of given video sequences and displaying the composite video sequence on a display device in communication with a computer processor, wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the method comprising the steps of:

(a) forming a background representation using the computer processor, said background representation including a common environment, wherein for each video sequence a weighted mask sequence is computed, said weighted mask sequence describing the probability of a pixel to belong to the background;

(c) aligning said plurality of given video sequences using the computer processor, based on a matching in size and position of elements from a background and/or from a foreground which are present in at least two of the plurality of given video sequences;

(d) forming the composite video sequence, using the computer processor, based on the formed background representation and the aligned video sequences; and

(e) displaying the composite video sequence on the display device.

54. A computer method for a computer processor to generate a composite video sequence from a plurality of given video sequences wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the method comprising the steps of:

(a) forming a background representation including a common environment, using the computer processor;

(b) ascertaining the representation of the respective one of the plurality of elements in each of the given video sequences, using the computer processor;

(c) coordinating the actions of the ascertained representations, using the computer processor, wherein said coordinating includes a spatial normalization between the given video sequences, for matching as to size and position a first visual element in one of the given video sequences with a second visual element in another of the given video sequences, wherein said visual elements are background elements;

(d) forming the composite video sequence based on the formed background representation and the coordinated representations, using the computer processor, such that the coordinated representations appear in the common environment; and

(e) displaying the composite video sequence on a display device.