Apparatus and method for spatio-temporal alignment of image sequences

• US 7,428,345 B2
• Filed: 05/10/2001
• Issued: 09/23/2008
• Est. Priority Date: 05/11/2000
• Status: Expired due to Fees

• Current Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science)
• Original Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science)
• Inventors: Irani, Michal, Caspi, Yaron
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US09/852,891
• Publication Number:
  US 20020094135A1
• Time in Patent Office: 2,693 Days
• Field of Search: 382/236, 382284-286, 382/287, 382/289, 382294-296, 382/276, 345/629, 345648-649, 345/651, 345/653, 345/723, 348/584, 348/513, 348/36, 463/31
• US Class Current: 382/294
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: H04N 5/2625 for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect sequence generated by event triggered capturing H04N7/188

Abstract

A method and system for aligning in at least one of time and space temporally ordered sequences of images comprising receiving a plurality of sequences of images, each sequence including a multiplicity of images, each plurality of sequences defining a space-time volume, and providing an output indication relating at least one point in a space-time volume corresponding to one of the plurality of sequences to at least one point in a space-time volume corresponding to at least another one of the plurality of sequences.

Claims

1. A method for aligning in at least one of time and space temporally ordered sequences of images comprising:

receiving a plurality of sequences of images, each sequence containing a multiplicity of images, each of said plurality of sequences defining a space-time volume, without requiring that images in said plurality of sequences be coincident in time;

providing an output indication relating at least one point in a space-time volume corresponding to one of said plurality of sequences to at least one point in a space-time volume corresponding to at least another one of said plurality of sequences thereby aligning said temporally orderes sequences of images.

2. A method for aligning according to claim 1 and wherein said providing an output indication comprises:

finding at least one global alignment in at least one of time and space between at least a portion of a first space-time volume corresponding to one of said plurality of sequences and at least a portion of a second space-time volume corresponding to at least another one of said plurality of sequences, which said at least one global alignment matches spatial-temporal variations occurring at plural locations in said first space-time volume to spatial-temporal variations occurring at plural locations in said second space-time volume.

3. A method for aligning according to claim 2 and wherein said finding performs correlation.

4. A method according to claim 2 wherein said step of finding at least one global alignment in at least one of time and space comprises finding at least one global alignment in time.

5. A method according to claim 2 wherein said step of finding at least one global alignment in at least one of time and space comprises finding at least one global alignment in space.

6. A method for aligning according to claim 1 and wherein said providing an output indication comprises:

finding at least one global alignment in at least one of time and space between at least a portion of a first space-time volume corresponding to one of said plurality of sequences and at least a portion of a second space-time volume corresponding to at least another one of said plurality of sequences, which said at least one global alignment matches at least temporal variations occurring at plural locations in said first space-time volume to at least temporal variations occurring at plural locations in said second space-time volume.

7. A method for aligning according to claim 6 and wherein said finding performs correlation.

8. A method according to claim 7 and wherein said temporal variations result from at least one of said sequences being at different frame rates, said sequence having a time shift therebetween and said sequences having a shift at sub-frame units.

9. A method according to claim 6 and wherein said temporal variations result from at least one of said sequences being at different frame rates, said sequence having a time shift therebetween and said sequences having a shift at sub-frame units.

10. A method according to claim 1 wherein said images comprise color images.

11. A method according to claim 1 and wherein said at least one point in a space-time volume comprises at least one of a point in a sub-frame unit and a point in a sub-pixel unit.

12. A system for aligning in at least one of time and space temporally ordered sequences of images comprising:

a space-time volume generator, receiving a plurality of sequences of images, each sequence containing a multiplicity of images, each of said plurality of sequences defining a space-time volume;

without requiring that images in said plurality of sequences be coincident in time; and

an aligner to align said temporally ordered sequences of images by providing an output indication relating at least one point in a space-time volume corresponding to one of said plurality of sequences to at least one point in a space-time volume corresponding to at least another one of said plurality of sequences.

13. A system for aligning according to claim 12 and wherein said aligner comprises a match-based global alignment finder, finding at least one global alignment in at least one of time and space between at least a portion of a first space-time volume corresponding to one of said plurality of sequences and at least a portion of a second space-time volume corresponding to at least another one of said plurality of sequences, which said at least one global alignment matches spatial-temporal variations occurring at plural locations in said first space-time volume to spatial-temporal variations occurring at plural locations in said second space-time volume.

14. A system for aligning according to claim 13 and wherein said match-based global alignment finder comprises a correlation based finder.

15. A system for aligning according to claim 12 and wherein said aligner comprises a match-based global alignment finder, finding at least one global alignment in at least one of time and space between at least a portion of a first space-time volume corresponding to one of said plurality of sequences and at least a portion of a second space-time volume corresponding to at least another one of said plurality of sequences, which said at least one global alignment matches at least temporal variations occurring at plural locations in said first space-time volume to spatial-temporal variations occurring at plural locations in said second space-time volume.

16. A system for aligning according to claim 15 and wherein said match-based global alignment finder comprises a correlation based finder.

17. A system for aligning according to claim 16 and wherein said temporal variations result from at least one of said sequences being at different frame rates, said sequences having a time shift therebetween and said sequences having a shift at sub-frame units.

18. A system for aligning according to claim 15 and wherein said temporal variations result from at least one of said sequences being at different frame rates, said sequences having a time shift therebetween and said sequences having a shift at sub-frame units.

19. A system for aligning according to claim 12 and wherein said at least one point in a space-time volume comprises at least one of a point in a sub-frame unit and a point in a sub-pixel unit.