Apparatus and method for alignment of spatial or temporal non-overlapping images sequences

• US 7,596,281 B2
• Filed: 12/04/2001
• Issued: 09/29/2009
• Est. Priority Date: 12/05/2000
• Status: Expired due to Fees

• Current Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science), Ydea Research And Development
• Original Assignee: Yeda Research and Development Co. Ltd. (Weizmann Institute of Science)
• Inventors: Irani, Michal, Caspi, Yaron
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US10/433,526
• Publication Number:
  US 20040071367A1
• Time in Patent Office: 2,856 Days
• Field of Search: 382/276, 382/294, 382/300, 382/312, 382/291, 382/295, 382/296, 382/284, 348/47, 348/48, 348/416, 348/513, 345/648, 345/672, 345/677, 375/240.18, 709/231
• US Class Current: 382/276
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/38 Registration of image sequences

Abstract

A method and system for computing at least one of spatial and temporal relationships between at least first and second sequences of representations having respective first and second temporal progressions and including employing the first and second temporal progressions to obtain at least one of the spatial and temporal relationships.

Claims

1. A method implemented on a computing device for computing at least one of spatial and temporal relationships between at least first and second sequences of visual representations, wherein at least a spatial relationship between said sequences is unknown, the method comprising:

for said first and second sequences of visual representations, computing on a computing device first and second sequences of ordered intra-sequence representation-to-representation transformations, respectively; and

employing at least a subset of said first and second sequences of transformations to compute on a computing device at least one of said spatial and temporal relationships for joint processing of said first and second sequences of visual representations, wherein said temporal relationship is computed from said at least a subset of sequences of transformations and wherein said spatial relationship is computed from said temporal relationship and said at least a subset of sequences of transformations.

2. A method according to claim 1 and wherein said visual representations comprise images.

3. A method according to claim 1 and wherein said representations comprise at least three-dimensional objects.

4. A method according to claim 1 and wherein said employing obtains said spatial relationship.

5. A method according to claim 1 and wherein said employing obtains said temporal relationship.

6. A method according to claim 1 and wherein said employing obtains said spatial and temporal relationship.

7. A method according to claim 1 and wherein said employing obtains said at least one of spatial and temporal relationships in the absence of a spatial overlap between said sequences of visual representations.

8. A method according to claim 1 and wherein said employing obtains said at least one of spatial and temporal relationships in the absence of common spatial information between said sequences of visual representations.

9. A method according to claim 1 and wherein said employing obtains said at least one of spatial and temporal relationships in the absence of common features between individual ones of said representations belonging to different ones of said at least first and second sequences of visual representations.

10. A method according to claim 1 and wherein said spatial relationship comprises a geometric transformation between said at least first and second sequences of representations.

11. A method according to claim 10 and wherein said spatial relationship comprises at least one of the following:

at least one parameter of said geometric transformation, a projective transformation, a homogranhy, a fundamental matrix and a non-parametric transformation.

12. A method according to claim 1 and wherein said temporal relationship comprises a temporal transformation between said at least first and second sequences of representations.

13. A method according to claim 12 and wherein said temporal relationship comprises at least one of the following:

at least one parameter of said temporal transformation, a time shift between said at least first and second sequences of representations and an affine transformation in time.

14. A method according to claim 1 and wherein at least some of said intra-sequence representation-to-representation transformations result from relative motion between sensors and a scene.

15. A method according to claim 14 and wherein said intra-sequence representation-to-representation transformations include motions taking place at least partially at different times.

16. A method according to claim 1 and wherein said intra-sequence representation-to-representation transformations include at least one of:

projective transformations, homographies, fundamental matrices, parametric transformations and non-parametric transformations.

17. A method according to claim 1 and wherein said employing comprises at least one of the following steps:

correlating said first and second sequences of transformations;

equating properties, up to a scale factor, of said first sequence of transformations and said second sequence of transformations;

comparing properties of said first sequence of transformations and said second sequence of transformations; and

correlating properties of said first sequence of transformations and said second sequence of transformations.

18. A method according to claim 1 and wherein said spatial relationship results from an acquisition relationship between first and second sensors acquiring respective said at least first and second sequences of visual representations, said relationship being one of:

fixed over time and changing in a known way over time.

19. A method according to claim 1 and wherein said at least first and second sequences of visual representations are one of the following:

acquired generally at the same time;

acquired generally at different times;

represent measurements from the same scene;

represent measurements from different portions of the same scene; and

represent measurements from different scenes.

20. A method according to claim 19 and wherein said measurements are generally the same for each sensor.

21. A method according to claim 19 and wherein said measurements are generally different for each sensor.

22. A method according to claim 1 and also comprising employing at least one of said spatial and temporal relationships for at least one of:

generating wide-screen movies from multiple at least partially non-overlapping narrow field of view movies, fusing images, integrating information contained in said at least first and second sequences of representations, comparing information contained in said at least first and second sequences of representations, and finding differences between information contained in said at least first and second sequences of representations.

23. A method according to claim 1 and also comprising employing at least one of said spatial and temporal relationships for alignment of images obtained at different zooms.

24. A method according to claim 1 and also comprising employing at least one of said spatial and temporal relationships for integrating information contained in said at least first and second sequences of representations and thereby providing an information output which exceeds the optical limitations of individual sensors.

25. A method according to claim 24 and wherein said optical limitations comprise at least one of:

limited spatial resolution, limited temporal resolution, limited spectral range, limited dynamic range, limited depth of focus, limited zoom, limited field of view, and limited view point.

26. A method according to claim 1 and wherein at least some of said intra-sequence representation-to-representation transformations comprise frame-to-frame transformations.

27. A system for computing at least one of spatial and temporal relationships between at least first and second sequences of visual representations, wherein at least a spatial relationship between said sequences is unknown, the system comprising:

input functionality for receiving at least one of said first sequence of representations and at least one of said second sequence of representations and for computing first and second sequences of ordered intra-sequence representation-to-representation transformations for said first and second sequences of visual representations, respectively; and

computation functionality employing at least a subset of said first and second sequences of transformations to compute at least one of said spatial and temporal relationships for joint processing of said first and second sequences of visual representations, wherein said temporal relationship is computed from said at least a subset of sequences of transformations and wherein said spatial relationship is computed from said temporal relationship and said at least a subset of sequences of transformations.

28. A system according to claim 27 and wherein said visual representations comprise images.

29. A system according to claim 27 and wherein said representations comprise at least three-dimensional objects.

30. A system according to claim 27 and wherein said computation functionality includes functionality for obtaining said spatial relationship.

31. A system according to claim 27 and wherein said computation functionality includes functionality for obtaining said temporal relationship.

32. A system according to claim 27 and wherein said computation functionality includes functionality for obtaining said spatial and temporal relationship.

33. A system according to claim 27 and wherein said computation functionality includes functionality for obtaining said at least one of spatial and temporal relationships in the absence of a spatial overlap between said sequences of visual representations.

34. A system according to claim 27 and wherein said -computation functionality includes functionality for obtaining said at least one of spatial and temporal relationships in the absence of common spatial information between said sequences of visual representations.

35. A system according to claim 27 and wherein said computation functionality includes functionality for obtaining said at least one of spatial and temporal relationships in the absence of common features between individual ones of said representations belonging to different ones of said at least first and second sequences of visual representations.

36. A system according to claim 27 and wherein said spatial relationship comprises a geometric transformation between said at least first and second sequences of representations.

37. A system according to claim 36 and wherein said spatial relationship comprises at least one of the following:

at least one parameter of said geometric transformation, a projective transformation, a homography, a fundamental matrix and a non-parametric transformation.

38. A system according to claim 27 and wherein said temporal relationship comprises a temporal transformation between said at least first and second sequences of representations.

39. A system according to claim 38 and wherein said temporal relationship comprises at least one of the following:

at least one parameter of said temporal transformation, a time shift between said at least first and second sequences of representations and an affine transformation in time.

40. A system according to claim 27 and wherein at least some of said intra-sequence representation-to-representation transformations result from relative motion between sensors and a scene.

41. A system according to claim 40 and wherein said intra-sequence representation-to-representation transformations include motions taking place at least partially at different times.

42. A system according to claim 27 and wherein said intra-sequence representation-to-representation transformations include at least one of:

projective transformations, homographies, fundamental matrices, parametric transformations and non-parametric transformations.

43. A system according to claim 27 and wherein said computation functionality comprises at least one of the following:

correlating said first and second sequences of transformations;

equating properties, up to a scale factor, of said first sequence of transformations and said second sequence of transformations;

comparing properties of said first sequence of transformations and said second sequence of transformations; and

correlating properties of said first sequence of transformations and said second sequence of transformations.

44. A system according to claim 27 and wherein said spatial relationship results from an acquisition relationship between first and second sensors acquiring respective said at least first and second sequences of visual representations, said relationship being one of:

fixed over time and changing in a known way over time.

45. A system according to claim 27 and wherein said at least first and second sequences of visual representations are one of the following:

acquired generally at the same time;

acquired generally at different times;

represent measurements from the same scene;

represent measurements from different portions of the same scene; and

represent measurements from different scenes.

46. A system according to claim 45 and wherein said measurements are generally the same for each sensor.

47. A system according to claim 45 and wherein said measurements are generally different for each sensor.

48. A system according to claim 27 and wherein said computation functionality includes functionality for employing at least one of said spatial and temporal relationships for at least of:

generating wide-screen movies from multiple at least partially non-overlapping narrow field of view movies, fusing images, integrating information contained in said at least first and second sequences of representations, comparing information contained in said at least first and second sequences of representations, and finding differences between information contained in said at least first and second sequences of representations.

49. A system according to claim 27 and wherein said computation functionality includes functionality for employing at least one of said spatial and temporal relationships for alignment of images obtained at different zooms.

50. A system according to claim 27 and wherein said computation functionality at least one of said spatial and temporal relationships for integrating information contained in said at least first and second sequences of representations and thereby providing an information output which exceeds the optical limitations of individual sensors.

51. A system according to claim 50 and wherein said optical limitations comprise at least one of:

limited spatial resolution, limited temporal resolution, limited spectral range, limited dynamic range, limited depth of focus, limited zoom, limited field of view, and limited view point.

52. A system according to claim 27 and wherein at least some of said intra-sequence representation-to-representation transformations comprise frame-to-frame transformations.