System and method for manipulating the point of interest in a sequence of images

• US 7,106,361 B2
• Filed: 02/12/2002
• Issued: 09/12/2006
• Est. Priority Date: 02/12/2001
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Kanade, Takeo, Collins, Robert
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): CZEKAJ, DAVID J
• Application Number:
  US10/074,313
• Publication Number:
  US 20020145660A1
• Time in Patent Office: 1,673 Days
• Field of Search: 348 36- 37, 348/135, 348/139, 348141-143, 348151-159, 348169-172, 348/218.1
• US Class Current: 348/159
• CPC Class Codes: G03B 41/00 Special techniques not covered by groups G03B31/00 - G03B39/00; Apparatus therefor
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A method and a system of generating a video image sequence. According to one embodiment, the method includes positioning a plurality of camera systems relative to a scene such that the camera systems define a gross trajectory. The method further includes transforming images from the camera systems to superimpose a secondary induced motion on the gross trajectory. And the method includes displaying the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.

Claims

1. A method of generating a video image sequence, comprising:

positioning a plurality camera systems relative to a scene such that the camera systems define a gross trajectory;

capturing images with the camera systems using parameters from a first of the camera systems and mapping data for the camera systems to compute parameters for the remainder of the camera systems;

applying a 2D projective image transformation to certain captured images from the camera systems to superimpose a secondary induced motion on the gross trajectory, wherein the image transformation is done independently of the three-dimensional structure of the scene; and

displaying the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory,

wherein the mapping data includes;

data regarding the geometric relationship of the camera systems to the scene;

data regarding the relationship between the zoom and the angular field of view for each camera system; and

data regarding the relationship between the focus and the depth of field for each camera system.

2. The method of claim 1, wherein positioning the plurality of camera systems includes positioning a plurality of pan/tilt camera systems relative to the scene.

3. The method of claim 1, wherein positioning the plurality of camera systems includes positioning a plurality of static camera systems relative to the scene.

4. The method of claim 1, wherein positioning the plurality of camera systems includes positioning at least one pan/tilt camera system and one static camera system relative to the scene.

5. The method of claim 1, wherein applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a one point correspondence between the images.

6. The method of claim 5, wherein applying the 2D image transformation according to a homography defined by a one point correspondence between the images includes mapping a point of interest in each image to the center of the image.

7. The method of claim 1, wherein applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a two point correspondence between the images.

8. The method of claim 7, wherein applying the 2D image transformation according to a homography defined by a two point correspondence between the images includes:

mapping a point of interest in each image to the center of the image; and

mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.

9. The method of claim 1, wherein applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a three point correspondence between the images.

10. The method of claim 9, wherein applying the 2D image transformation according to a homography defined by a three point correspondence between the images includes:

mapping a translation point in each image to the center of the image;

mapping a point of interest in each image to the translation point in each image; and

mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.

11. The method of claim 1, further comprising:

generating an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and

displaying the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.

12. The method of claim 11, wherein generating the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems includes generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.

13. The method of claim 1, wherein positioning the plurality of camera systems relative to the scene includes positioning the plurality of camera systems in a close-ended configuration relative to the scene.

14. The method of claim 1, wherein positioning the plurality of camera systems relative to the scene includes positioning the plurality of camera systems in an array configuration.

15. A system for generating a video image sequence of an object within a scene, comprising:

means for capturing an image from a plurality of camera systems positioned relative to the scene such that the camera systems define a gross trajectory, wherein parameters from a first of the camera systems and mapping data for the camera systems are used to compute parameters for the remainder of the camera systems; and

means for 2D projective image transforming certain captured images from the camera systems to superimpose a secondary induced motion on the gross trajectory, wherein the image transformation is done independently of the three-dimensional structure of the scene,

wherein the mapping data includes;

data regarding the geometric relationship of the camera systems to the scene;

data regarding the relationship between the zoom and the angular field of view for each camera system; and

data regarding the relationship between the focus and the depth of field for each camera system.

16. The system of claim 15, further comprising means for controlling the plurality of camera systems such that the camera systems are simultaneously aimed at a target within the scene and a size of the target in the images from the camera systems is substantially the same over time.

17. The system of claim 16, further comprising means for outputting the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.

18. The system of claim 15, wherein the means for applying the 2D image transformation includes means for applying a 2D image transformation according to a homography defined by a one-point correspondence between the images.

19. The system of claim 18, wherein the means for applying the 2D image transformation according to a homography defined by a one point correspondence between the images includes means for mapping a point of interest in each image to the center of the image.

20. The system of claim 19, wherein the point of interest is not a point of the target.

21. The system of claim 19, wherein the point of interest is a point of the target.

22. The system of claim 15, wherein the means for applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a two-point correspondence between the images.

23. The system of claim 22, wherein the means for applying the 2D image transformation according to a homography defined by a two-point correspondence between the images includes:

means for mapping a point of interest in each image to the center of the image; and

means for mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.

24. The system of claim 15, wherein the means for applying the 2D image transformation includes means for applying a 2D image transformation according to a homography defined by a three-point correspondence between the images.

25. The system of claim 24, wherein the means for applying the 2D image transformation according to a homography defined by a three-point correspondence between the images includes:

means for mapping a translation point in each image to the center of the image;

means for mapping a point of interest in each image to the translation point in each image; and

means for mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.

26. The system of claim 15, further comprising:

means for generating an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and

means for outputting the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.

27. The system of claim 26, wherein the means for generating the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems includes means for generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.

28. A system for generating a video image sequence of an object within a scene, comprising:

a plurality of camera systems positioned relative to the scene such that the camera systems define a gross trajectory;

a video storage unit in communication with the camera systems for storing images captured by the camera systems, wherein parameters from a first of the camera systems and mapping data for the camera systems are used to compute parameters for the remainder of the camera systems; and

a frame-sequencing module in communication with the video storage unit for applying a 2D projective transformation to certain of the captured images of the camera systems retrieved from the video storage unit to superimpose a secondary induced motion on the gross trajectory, wherein the image transformation is done independently of the three-dimensional structure of the scenes,

wherein the mapping data includes;

data regarding the geometric relationship of the camera systems to the scene;

data regarding the relationship between the zoom and the angular field of view for each camera system; and

data regarding the relationship between the focus and the depth of field for each camera system.

29. The system of claim 28, further comprising means for controlling the plurality of camera systems such that the camera systems are simultaneously aimed a target within the scene and a size of the target in the images from the camera systems is substantially the same over time.

30. The system of claim 29, wherein the frame-sequencing module is further for outputting the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.

31. The system of claim 30, wherein the frame-sequencing module is further for:

generating an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and

outputting the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.

32. The system of claim 31, wherein the frame-sequencing module is for generating the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems by generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.

33. The system of claim 28, wherein the 2D image transformation includes a 2D image transformation according to a homography defined by a one-point correspondence between the images.

34. The system of claim 33, wherein frame-sequencing module is for applying the 2D image transformation according to a homography defined by a one point correspondence between the images includes by mapping a point of interest in each image to the center of the image.

35. The system of claim 34, wherein the point of interest is not a point of the target.

36. The system of claim 34, wherein the point of interest is a point of the target.

37. The system of claim 28, wherein the 2D image transformation includes a 2D image transformation according to a homography defined by a two-point correspondence between the images.

38. The system of claim 37, wherein the frame-sequencing module is for applying the 2D image transformation according to a homography defined by a two-point correspondence between the images by:

mapping a point of interest in each image to the center of the image; and

mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.

39. The system of claim 28, wherein the 2D image transformation includes a 2D image transformation according to a homography defined by a three-point correspondence between the images.

40. The system of claim 39, wherein the frame-sequencing module is for applying the 2D image transformation according to a homography defined by a three-point correspondence between the images by:

mapping a translation point in each image to the center of the image;

mapping a point of interest in each image to the translation point in each image; and

mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.

41. A computer readable medium, having stored thereon instructions which, when executed by a processor, cause the processor to:

compute parameters for a plurality of camera systems based on the parameters for one of the camera systems and based on mapping data for the camera systems, wherein the camera systems are positioned relative to a scene to define a gross trajectory to superimpose a secondary induced motion on the gross trajectory;

apply a 2D projective transformation to certain images captured by a plurality of camera systems positioned relative to a scene to define a gross trajectory to superimpose a secondary induced motion on the gross trajectory, wherein the image transformation is done independently of the three-dimensional structure of the scene; and

output the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory,

wherein the mapping data includes;

data regarding the geometric relationship of the camera systems to the scene;

data regarding the relationship between the zoom and the angular field of view for each camera system; and

data regarding the relationship between the focus and the depth of field for each camera system.

42. The computer readable medium of claim 41, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation by applying a 2D image transformation according to a homography defined by a one point correspondence between the images.

43. The computer readable medium of claim 42, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation according to a homography defined by a one point correspondence between the images by mapping a point of interest in each image to the center of the image.

44. The computer readable medium of claim 41, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation by applying a 2D image transformation according to a homography defined by a two point correspondence between the images.

45. The computer readable medium of claim 44, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation according to a homography defined by a two point correspondence between the images by

mapping a point of interest in each image to the center of the image;

and

mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.

46. The computer readable medium of claim 41, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation includes applying a 2D image transformation according to a homography defined by a three point correspondence between the images.

47. The computer readable medium of claim 46, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation according to a homography defined by a three point correspondence between the images by:

mapping a translation point in each image to the center of the image;

mapping a point of interest in each image to the translation point in each image; and

mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.

48. The computer readable medium of claim 41, having further stored thereon instructions which, when executed by the processor, cause the processor to:

generate an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and

display the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.

49. The computer readable medium of claim 48, having further stored thereon instructions which, when executed by the processor, cause the processor to generate the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems by generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.