Method and system for combining video sequences with spatio-temporal alignment

• US 7,843,510 B1
• Filed: 01/19/1999
• Issued: 11/30/2010
• Est. Priority Date: 01/16/1998
• Status: Expired due to Fees

• Current Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Original Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Inventors: Ayer, Serge, Vetterli, Martin
• Primary Examiner(s): LEE, MICHAEL
• Application Number:
  US09/234,036
• Time in Patent Office: 4,333 Days
• Field of Search: 348584-600, 348/157, 348/159, 348/512, 348/513, 348/564, 348/565
• US Class Current: 348/584
• CPC Class Codes: G07C 1/24 Race time-recorders
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Given two video sequences, a composite video sequence can be generated which includes visual elements from each of the given sequences, suitably synchronized and represented in a chosen focal plane. For example, given two video sequences with each showing a different contestant individually racing the same down-hill course, the composite sequence can include elements from each of the given sequences to show the contestants as if racing simultaneously. A composite video sequence can be made also by similarly combining a video sequence with an audio sequence.

Claims

1. A method for generating a composite video sequence from a plurality of given video sequences, comprising the steps of:

(a) synchronizing, using at least one of a workstation, a personal computer, and dedicated processing hardware, the given video sequences into a corresponding plurality of synchronized video sequences;

(b) choosing a camera reference coordinate system for each frame of each synchronized video sequence and obtaining, using at least one of a workstation, a personal computer, and dedicated processing hardware, a camera coordinate transformation between the camera reference coordinate system and the corresponding frame of each of the synchronized video sequences; and

(c) forming, using at least one of a workstation, a personal computer, and dedicated processing hardware, the composite video sequence from the synchronized video sequences by transforming each sequence based on the camera coordinate transformation into a chosen focal plane and by superimposing the transformed sequences for merged simultaneous visualization on a single display.

2. The method according to claim 1, wherein the camera coordinate transformation is estimated from the plurality of synchronized video sequences.

3. The method according to claim 1, wherein the camera coordinate transformation is obtained from recorded camera position data and parameters including focal length.

4. The method according to claim 1, wherein for each video sequence a respective foreground object and a background are distinguished, and wherein the transformed foreground objects are superimposed on the transformed background.

5. The method according to claim 4, further comprising the step of extracting, using at least one of a workstation, a personal computer, and dedicated processing hardware, the respective foreground object and the background for each of the synchronized video sequences.

6. The method according to claim 5, wherein extracting comprises producing a weight mask sequence, with each weight mask being an array having an entry for each pixel position for differentiating between the respective foreground object and the background.

7. The method according to claim 4, wherein synchronizing is with respect to a timed event in the given sequences.

8. The method according to claim 1, wherein synchronizing is with respect to a common spatial event in the given sequences.

9. The method according to claim 1, wherein synchronizing is with respect to two events in each of the given sequences, with time scaling for equalizing time between the events.

10. The method according to claim 1, wherein the chosen focal plane corresponds to the focal plane of one of the given sequences, and wherein the composite sequence is as viewed from the camera location of the one of the given sequences.

11. The method according to claim 1, wherein forming the composite sequences is on a frame-by-frame basis.

12. The method according to claim 1, wherein forming the composite sequence is based on several frames of at least one of the sequences, for an expanded field of view in the composite sequence as compared with the one of the sequences.

13. The method according to claim 1, wherein the given video sequences are from a sports event.

14. The method according to claim 13, wherein the sports event is a ski race.

15. The method according to claim 13, wherein the sports event is a car race.

16. The method according to claim 1, wherein the given video sequences have biomedical significance.

17. The method according to claim 16, wherein biomedical significance comprises significance as to movement of a limb of a patient.

18. The method according to claim 1, wherein the given video sequences comprise car crash test sequences.

19. The method according to claim 18, wherein the car crash test sequences comprise images of cars being tested.

20. The method according to claim 18, wherein the car crash test sequences comprise images of crash dummies in cars being tested.

21. A system for generating a composite video sequence from a plurality of given video sequences, comprising:

(a) means for synchronizing the given video sequences into a corresponding plurality of synchronized video sequences;

(b) means for choosing a camera reference coordinate system for each frame of each synchronized video sequence and obtaining a camera coordinate transformation between the camera reference coordinate system and the corresponding frame of each of the synchronized video sequences; and

(c) means for forming the composite video sequence from the synchronized video sequences by transforming each sequence based on the camera coordinate transformation into a chosen focal plane and by superimposing the transformed sequences for merged simultaneous visualization on a single display.

22. A method for determining differential time between two contestants at a specified location in a race, comprising:

(a) synchronizing, using at least one of a workstation, a personal computer, and dedicated processing hardware, a video sequence of one of the contestants with a video sequence of the other contestant;

(b) choosing a camera reference coordinate system for each frame of each synchronized video sequence and obtaining, using at least one of a workstation, a personal computer, and dedicated processing hardware, a camera coordinate transformation between the camera reference coordinate system and the corresponding frame of each of the synchronized video sequences;

(c) forming, using at least one of a workstation, a personal computer, and dedicated processing hardware, a composite video sequence from the synchronized video sequences by transforming each sequence based on the camera coordinate transformation into a chosen focal plane and by superimposing the transformed sequences for merged simultaneous visualization on a single display; and

(d) counting, using at least one of a workstation, a personal computer, and dedicated processing hardware, the number of frames between the contestants passing the location in the race.

23. A broadcast service for transmitting a composite video sequence whose generation comprises the steps of:

(a) synchronizing, using at least one of a workstation, a personal computer, and dedicated processing hardware, a plurality of given video sequences into a corresponding plurality of synchronized video sequences;

(b) choosing a camera reference coordinate system for each frame of each synchronized video sequence and obtaining, using at least one of a workstation, a personal computer, and dedicated processing hardware, a camera coordinate transformation between the camera reference coordinate system and the corresponding frame of each of the synchronized video sequences; and

(c) forming, using at least one of a workstation, a personal computer, and dedicated processing hardware, the composite video sequence from the synchronized video sequences by transforming each sequence based on the camera coordinate transformation into a chosen focal plane and by superimposing the transformed sequences for merged simultaneous visualization on a single display.