Method and system for estimating global motion in video sequences

• US 20060023786A1
• Filed: 11/18/2003
• Published: 02/02/2006
• Est. Priority Date: 11/26/2002
• Status: Active Grant

• Current Assignee: British Telecommunications PLC (BT Group PLC)
• Original Assignee: British Telecommunications PLC (BT Group PLC)
• Inventors: Xu, Li-Qun, Morrison, David, Li, Yongmin, Nightngale, Charles
• Granted Patent: US 8,290,040 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: G06T 3/16 Spatio-temporal transformations, e.g. video cubism
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Abstract

The invention relates to estimating the global motion between frames of a motion-compensated inter-frame encoded video sequence, directly from the motion vectors encoded within the frames. For any particular frame, a motion estimation is determined from motion vectors direct from the frame'"'"'s anchor frame to the frame in question. This motion estimation is then checked against pre-determined criteria, and where the criteria are not met, re-estimation along a different route is performed, using the bi-directional motion vectors contained within B-frames. A panoramic image generating method and system which makes uses of the global motion estimations thus obtained is also described.

Claims

1. A method of global motion estimation between frames of a motion-compensated inter-frame encoded video sequence, each frame of the sequence having a plurality of motion vectors encoded therein relating the frame to a preceding and/or succeeding frame of the sequence, the method comprising, for a particular frame:

a) determining a motion estimation representative of the global motion between the particular frame and its anchor frame on the basis of motion vectors therebetween;

b) determining one or more further motion estimations representative of the global motion between the particular frame and its anchor frame at least partially on the basis of motion vectors between the particular frame and one or more preceding or succeeding other frames; and

c) selecting one of the motion estimations which meets at least one predetermined criterion as being representative of the global motion of the frame.

2. A method according to claim 1, wherein the determining step b) further comprises the steps of:

e) determining one or more motion estimations representative of the global motion of the frame with respect to one or more respective preceding or succeeding other frames;

f) determining one or motion estimations respectively representative of the global motion of the one or more other frames with respect to the anchor frame; and

g) accumulating the respective motion estimations to give one or more respective overall motion estimations each substantially representative of the global motion of the frame with respect to the anchor frame.

3. A method according to claims 1, wherein the selecting step c) further comprises testing the motion estimations in turn;

and outputting a motion estimation as being representative of the global motion of the frame if it passes the test, wherein the test is applied in turn to motion estimations once they have been determined, and if the test is passed then no further motion estimations are determined.

4. A method according to claim 3, wherein the test comprises comparing the motion estimation with a threshold value, wherein the test is passed if the parameters of the motion estimation do not exceed the threshold value.

5. A method according to any of claims 3, wherein if the test is failed, the method further comprises:

interpolating between the motion estimations of adjacent frames to give an interpolated motion estimation which is then output as the motion estimation representative of the global motion of the frame.

6. A method of generating panoramic images from a motion-compensated inter-frame encoded video sequence, the method comprising:

for each frame of the sequence, determining the global motion of each frame with respect to its anchor frame using the method of any of the preceding claims; and

generating at least one panoramic image representing the frames of the video sequence using the global motion estimations thus determined.

7. A method according to claim 6, wherein the generating step further comprises:

selecting a particular frame of the sequence as a reference frame, the plane of the reference frame being a reference plane;

for each frame other than the reference frame, accumulating the global motion estimations from each frame back to the reference frame;

warping each frame other than the reference frame onto the reference plane using the accumulated global motion estimations to give one or more pixel values for each pixel position in the reference plane; and

for each pixel position in the reference plane, selecting one of the available pixel values for use as the pixel value in the panoramic image.

8. A method according to claim 7, wherein the selecting step comprises selecting a substantially median pixel value from the available pixel values for use in a background panoramic image.

9. A method according to claim 7, wherein the selecting step comprises selecting a substantially most different pixel value from the available pixel values for use in a foreground panoramic image.

10. A method according to claim 7, wherein the selecting step comprises:

calculating the mean pixel value of the available pixel values;

calculating the L1 distance between each available pixel value and the calculated mean pixel value; and

select the pixel value with the median L1 distance for use in a background panoramic image.

11. A method according to claim 7, wherein the selecting step comprises:

calculating the mean pixel value of the available pixel values;

calculating the L1 distance between each available pixel value and the calculated mean pixel value; and

select the pixel value with the maximum L1 distance for use in a foreground panoramic image.

12. A computer program or suite of programs arranged such that when executed on a computer system the program or suite of programs causes the computer system to perform the method of claim 1.

13. A computer readable storage medium storing a computer program or suite of programs according to claim 12.

14. A system for global motion estimation between frames of a motion-compensated inter-frame encoded video sequence, each frame of the sequence having a plurality of motion vectors encoded therein relating the frame to a preceding and/or succeeding frame of the sequence, the system comprising:

video processing means arranged in use to;

i) determine a motion estimation representative of the global motion between the particular frame and its anchor frame on the basis of motion vectors therebetween;

ii) determine one or more further motion estimations representative of the global motion between the particular frame and its anchor frame at least partially on the basis of motion vectors between the particular frame and one or more preceding or succeeding other frames; and

a motion estimation selector means arranged in use to select one of the motion estimations which meets at least one predetermined criterion as being representative of the global motion of the frame.

15. A system according to claim 14, wherein the video processing means is further arranged in use to:

determine one or more motion estimations representative of the global motion of the frame with respect to a one or more respective preceding or succeeding other frames;

determine one or motion estimations respectively representative of the global motion of the one or more other frames with respect to the anchor frame; and

accumulate the respective motion estimations to give one or more respective overall motion estimations each substantially representative of the global motion of the frame with respect to the anchor frame.

16. A method according to claim 14, wherein the motion estimation selector means further comprises motion estimation testing means for testing the motion estimations in turn, and providing an indication as to whether the test is passed;

and output means arranged in use to output a motion estimation as being representative of the global motion of the frame in dependence on the receipt of an indication that the test was passed from the motion estimation testing means, wherein the motion estimation testing means is further arranged to apply the test in turn to motion estimations once they have been determined, and if the test is passed then no further motion estimations are determined.

17. A system according to claim 16, wherein the test comprises comparing the motion estimation with a threshold value, wherein the test is passed if the parameters of the motion estimation do not exceed the threshold value.

18. A system according to claim 16, and further comprising motion estimation interpolation means arranged to interpolate between the motion estimations of adjacent frames to give an interpolated motion estimation, in dependence on the motion estimation testing means indicating that the test has been failed by all the motion estimations determined by the video processing means.

19. A system for generating panoramic images from a motion-compensated inter- frame encoded video sequence, comprising:

a system for global motion estimation between frames of a motion-compensated inter-frame encoded video sequence as claimed in claim 14, and further arranged to provide global motion estimations for each frame; and

panoramic image generating means for generating at least one panoramic image representing the frames of the video sequence using the global motion estimations thus determined.

20. A system according to claim 19, wherein the panoramic image generating means is further arranged in use to:

select a particular frame of the sequence as a reference frame, the plane of the reference frame thereby being a reference plane;

for each frame other than the reference frame, accumulate the global motion estimations from each frame back to the reference frame;

warp each frame other than the reference frame onto the reference plane using the accumulated global motion estimations to give one or more pixel values for each pixel in the reference plane; and

for each pixel position in the reference plane, select one of the available pixel values for use as the pixel value in the panoramic image.

21. A system according to claim 20, wherein the panoramic image generating means is further arranged to select a substantially median pixel value from the available pixel values for use in a background panoramic image.

22. A system according to claim 19, wherein the panoramic image generating means is further arranged to select a substantially most different pixel value from the available pixel values for use in a foreground panoramic image.

23. A system according to claim 19, wherein the panoramic image generating means is further arranged to:

calculate the mean pixel value of the available pixel values;

calculate the L1 distance between each available pixel value and the calculated mean pixel value; and

select the pixel value with the median L1 distance for use in a background panoramic image.

24. A system according to claim 19, wherein the panoramic image generating means is further arranged to:

calculate the mean pixel value of the available pixel values;

calculate the L1 distance between each available pixel value and the calculated mean pixel value; and

select the pixel value with the maximum L1 distance for use in a foreground panoramic image.