PROCESSING OF IMAGES TO REPRESENT A TRANSITION IN VIEWPOINT

• US 20110229054A1
• Filed: 07/23/2009
• Published: 09/22/2011
• Est. Priority Date: 07/23/2008
• Status: Active Grant

• Current Assignee: Grass Valley USA Incorporated
• Original Assignee: Snell Limited (Belden Inc.)
• Inventors: Weston, Martin, Pearson, James
• Granted Patent: US 8,515,205 B2
• Time in Patent Office: Days
• US Class Current: 382/276
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: H04N 5/265 Mixing

Abstract

When mixing cutting between video cameras viewing a common scene from different viewpoints, geometric transforms that vary from image to image are applied to one or both camera outputs so as to create an apparent point of view that moves along on a path joining the viewpoints of the cameras.

Claims

1. A method of making an output sequence of images representing a transition between two input images or image sequences where the two input images or sequences of images include views of a common scene from different viewpoints, comprising the steps of creating transformed images or image sequences having an apparent point of view that moves along on a path joining the viewpoints of the two input images or sequences of images, wherein geometric transforms that vary over the said output sequence are applied to at least some input images or sequences of images so as to create said respective transformed images or image sequences and including the transformed images or sequences of images in the said output sequence of images.

2. A method according to claim 1 in which the said geometric transforms smoothly interpolate the apparent fields of view of the transformed images between the respective input image or image sequence fields of view.

3. A method according to claim 1 in which the said geometric transform is a perspective transform that dynamically transforms the shape of a notional laminar object lying in a plane in the said common scene so that it maintains the shape it would appear to have from the said apparent point of view.

4. A method according to claim 3 in which the said notional laminar object is aligned in the two transformed images or image sequences.

5. A method according to claim 3 in which the said transform dynamically transforms the shapes of a plurality of notional laminar objects lying in respective planes in the said common scene so that the said notional laminar objects maintain the respective shapes they would appear to have from the said apparent point of view.

6. A method according to claim 1 in which the field of view of a transformed image is augmented by including picture information from a stored reference image.

7. A method according to claim 6 in which the field of view of a transformed image is augmented by including picture information from a stored reference image only in those parts of the transformed image that were not included in the said transformed image prior to its transformation.

8. A method according to claim 6 in which the said stored reference image is transformed to the same viewpoint and field of view as the said transformed image.

9. A method according to claim 6 in which the said stored reference image is created by transforming image data from a plurality of cameras to a common position and field of view and combining the said transformed image data in a reference-image store.

10. A method according to claim 1 in which the height of the said path is chosen so that the height smoothly varies between the respective heights of the input image or image sequence viewpoints.

11. A method according to claim 3 in which the said path is chosen so that the distance between:

points on the said path and respective points on the said plane corresponding to the centres of the transformed images having an apparent point of view from that point on the path smoothly varies between the respective distances corresponding to the two ends of the said path.

12. A method according to claim 1 in which the said path is chosen so that the horizontal angle of the line joining the said apparent point of view to a point in the scene corresponding to a chosen position in the transformed image smoothly varies between the respective angles corresponding to the ends of the said path.

13. A method according to claim 1 in which the said path is derived automatically from data associated with the said input images or image sequences.

14. A method according to claim 1 in which the said geometric transform varies a magnification of a transformed image in dependence on the distance between the said apparent point of view and a point in the scene corresponding to a chosen position in the said transformed image.

15. A method according to claim 1 in which the said output sequence is formed by cross fading between the respective said transformed images or image sequences.

16. A method according to claim 1 in which the said output sequence is formed by non additively mixing between the respective said transformed images or image sequences.

17. A method according to claim 1 in which the said output sequence is formed by wiping between the respective said transformed images or image sequences.

18. A method according to claim 1 in which the said output sequence is formed by making a cut between the respective said transformed images or image sequences at a point on the said path.

19. A method according to claim 1 in which at least one of the said input image sequences comprises the forward or backward replay of a recording at a speed different from that at which it was recorded.

20. Apparatus for use in making an output sequence of images representing a transition between two input images or image sequences, comprising a mixer for receiving two input images or sequences of images from respective cameras viewing a common scene from different viewpoints, a path interpolator receiving camera calibration information and at least one digital video effects unit receiving images from a camera and path information from the path interpolator, wherein geometric transforms that vary over the said output sequence are applied to at least some input images or sequences of images so as to create respective transformed images or image sequences having an apparent point of view that moves along on a path joining the viewpoints of the two input images or sequences of images and the transformed images or sequences of images are included in the output sequence of images from the mixer.

21. Apparatus according to claim 20 in which the said geometric transforms smoothly interpolate the apparent fields of view of the transformed images between the respective input image or image sequence fields of view.

22. Apparatus according to claim 20 in which the said geometric transform is a perspective transform that dynamically transforms the shape of a notional laminar object lying in a plane in the said common scene so that it maintains the shape it would appear to have from the said apparent point of view.

23. Apparatus according to claim 20 in which the said notional laminar object is aligned in the two transformed images or image sequences.

24. Apparatus according to claim 20 in which the said transform dynamically transforms the shapes of a plurality of notional laminar objects lying in respective planes in the said common scene so that the said notional laminar objects maintain the respective shapes they would appear to have from the said apparent point of view.

25. Apparatus according to claim 20 in which the field of view of a transformed image is augmented by including picture information from a stored reference image in those parts of the transformed image that were not included in the said transformed image prior to its transformation.

26. Apparatus according to claim 25 in which the said stored reference image is transformed to the same viewpoint and field of view as the said transformed image.

27. Apparatus according to claim 25 in which the said stored reference image is created by transforming image data from a plurality of cameras to a common position and field of view and combining the said transformed image data in a reference-image store.

28. Apparatus according to claim 20 in which the height of the said path is chosen so that the height smoothly varies between the respective heights of the input image or image sequence viewpoints.

29. Apparatus according to claim 20 in which the said path is chosen so that the distance between:

points on the said path and respective points on the said plane corresponding to the centres of the transformed images having an apparent point of view from that point on the path smoothly varies between the respective distances corresponding to the two ends of the said path.

30. Apparatus according to claim 20 in which the said path is chosen so that the horizontal angle of the line joining the said apparent point of view to a point in the scene corresponding to a chosen position in the transformed image smoothly varies between the respective angles corresponding to the ends of the said path.

31. Apparatus according to claim 20 in which the said path is derived automatically from data associated with the said input images or image sequences.

32. Apparatus according to claim 20 in which the said geometric transform varies the magnification of a transformed image in dependence on the distance between the said apparent point of view and a point in the scene corresponding to a chosen position in the said transformed image.

33. Apparatus according to claim 20 in which the said output sequence is formed by cross fading between the respective said transformed images or image sequences.

34. Apparatus according to claim 20 in which the said output sequence is formed by non additively mixing between the respective said transformed images or image sequences.

35. Apparatus according to claim 20 in which the said output sequence is formed by wiping between the respective said transformed images or image sequences.

36. Apparatus according to claim 20 in which the said output sequence is formed by making a cut between the respective said transformed images or image sequences at a point on the said path.

37. Apparatus according to claim 20 in which at least one of the said input image sequences comprises the forward or backward replay of a recording at a speed different from that at which it was recorded.

38. A non-transitory computer program product containing instructions adapted to cause programmable apparatus to implement a method of making an output sequence of images representing a transition between two input images or image sequences where the two input images or sequences of images include views of a common scene from different viewpoints, comprising the steps of creating transformed images or image sequences having an apparent point of view that moves along on a path joining the viewpoints of the two input images or sequences of images, wherein geometric transforms that vary over the said output sequence are applied to at least some input images or sequences of images so as to create said respective transformed images or image sequences and including the transformed images or sequences of images in the said output sequence of images.