Method and system for estimating global motion in video sequences

• US 8,290,040 B2
• Filed: 11/18/2003
• Issued: 10/16/2012
• Est. Priority Date: 11/26/2002
• Status: Active Grant

• Current Assignee: British Telecommunications PLC (BT Group PLC)
• Original Assignee: British Telecommunications PLC (BT Group PLC)
• Inventors: Li, Yongmin, Xu, Li-Qun, Morrison, David G, Nightingale, Charles, Morphett, Jason
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): PONTIUS, JAMES M
• Application Number:
  US10/535,420
• Publication Number:
  US 20060023786A1
• Time in Patent Office: 3,255 Days
• Field of Search: 375/240.12, 375/240.16, 345/419, 345/629
• US Class Current: 375/240.12
• CPC Class Codes: G06T 3/16 Spatio-temporal transformations, e.g. video cubism
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Abstract

The invention relates to estimating the global motion between frames of a motion-compensated inter-frame encoded video sequence, directly from the motion vectors encoded within the frames. For any particular frame, a motion estimation is determined from motion vectors direct from the frame'"'"'s anchor frame to the frame in question. This motion estimation is then checked against pre-determined criteria, and where the criteria are not met, re-estimation along a different route is performed, using the bi-directional motion vectors contained within B-frames. A panoramic image generating method and system which makes uses of the global motion estimations thus obtained is also described.

Claims

1. A computer-implemented method of global motion estimation between frames of a motion-compensated inter-frame encoded video sequence, the method comprising, for a particular frame of the inter-frame encoded video sequence:

a) determining, using one or more computer processing systems, a motion estimation representative of the global motion between the particular frame and a first preceding or first succeeding frame of the inter-frame encoded video sequence, on the basis of motion vectors directly between the particular frame and the first preceding or first succeeding frame along a first route between the particular frame and the first preceding or first succeeding frame;

b) determining, using the one or more computer processing systems, one or more further motion estimations representative of the global motion between the particular frame and the first preceding or first succeeding frame at least partially on the basis of motion vectors between the particular frame and one or more preceding or succeeding other frames along one or more second routes between the particular frame and the first preceding or first succeeding frame, wherein the one or more second routes are different from the first route; and

c) selecting, using the one or more computer processing systems, one of the motion estimations which meets at least one predetermined criterion as being representative of the global motion of the particular frame;

d) storing or outputting said selected motion estimation for use in registering an image of said particular frame with an image of said first preceding or first succeeding frame.

2. A method according to claim 1, wherein the determining step b) further comprises the steps of:

e) determining one or more motion estimations representative of the global motion of the frame with respect to one or more respective preceding or succeeding other frames;

f) determining one or motion estimations respectively representative of the global motion of the one or more other frames with respect to the first preceding or first succeeding frame; and

g) accumulating the respective motion estimations to give one or more respective overall motion estimations each substantially representative of the global motion of the frame with respect to the first preceding or first succeeding frame.

3. A method according to claim 1, wherein

the selecting step c) further comprises testing the motion estimations in turn;

and outputting a motion estimation as being representative of the global motion of the frame if it passes the test, wherein the test is applied in turn to motion estimations once they have been determined, and if the test is passed then no further motion estimations are determined.

4. A method according to claim 3, wherein

the test comprises comparing the motion estimation with a threshold value, wherein the test is passed if the parameters of the motion estimation do not exceed the threshold value.

5. A method according to claim 3, wherein

if the test is failed, the method further comprises:

interpolating between the motion estimations of adjacent frames to give an interpolated motion estimation which is then output as the motion estimation representative of the global motion of the frame.

6. A method according to claim 1 further comprising

generating panoramic images from a motion-compensated inter-frame encoded video sequence, the generating comprising:

for each frame of the sequence, determining the global motion of each frame with respect to the first preceding or first succeeding frame using the method of claim 1; and

generating at least one panoramic image representing the frames of the video sequence using the frame global motion estimations thus determined.

7. A method according to claim 6, wherein

the generating step further comprises:

selecting a particular frame of the sequence as a reference frame, the plane of the reference frame being a reference plane;

for each frame other than the reference frame, accumulating the frame global motion estimations from each frame back to the reference frame;

warping each frame other than the reference frame onto the reference plane using the accumulated frame global motion estimations to give one or more pixel values for each pixel position in the reference plane; and

for each pixel position in the reference plane, selecting one of the available pixel values for use as the pixel value in the panoramic image.

8. A method according to claim 7, wherein

the selecting step comprises selecting a substantially median pixel value from the available pixel values for use in a background panoramic image.

9. A method according to claim 7, wherein

the selecting step comprises selecting a substantially most different pixel value from the available pixel values for use in a foreground panoramic image.

10. A method according to claim 7, wherein

the selecting step comprises:

calculating the mean pixel value of the available pixel values;

calculating the L1 distance between each available pixel value and the calculated mean pixel value; and

selecting the pixel value with the median L1 distance for use in a background panoramic image.

11. A method according to claim 7, wherein

the selecting step comprises:

calculating the mean pixel value of the available pixel values;

calculating the L1 distance between each available pixel value and the calculated mean pixel value; and

selecting the pixel value with the maximum L1 distance for use in a foreground panoramic image.

12. A non-transitory computer readable storage medium storing a computer program or suite of programs such that when executed on at least one computer system the program or suite of programs causes the at least one computer system to perform a method of global motion estimation between frames of a motion-compensated inter-frame encoded video sequence, the method comprising, for a particular frame of the inter-frame encoded video sequence:

a) determining a motion estimation representative of the global motion between the particular frame and a first preceding or first succeeding frame of the inter-frame encoded video sequence, on the basis of motion vectors directly between the particular frame and the first preceding or first succeeding frame along a first route between the particular frame and the first preceding or first succeeding frame;

b) determining one or more further motion estimations representative of the global motion between the particular frame and the first preceding or first succeeding frame at least partially on the basis of motion vectors between the particular frame and one or more preceding or succeeding other frames along one or more second routes between the particular frame and the first preceding or first succeeding frame, wherein the one or more second routes are different from the first route; and

c) selecting one of the motion estimations which meets at least one predetermined criterion as being representative of the global motion of the particular frame;

d) storing or outputting said selected motion estimation for use in registering an image of said particular frame with an image of said first preceding or first succeeding frame.

13. A system for global motion estimation between frames of a motion-compensated inter-frame encoded video sequence, the system comprising:

a video processor arranged in use to;

i) determine a motion estimation representative of the global motion directly between a particular frame of the inter-frame encoded video sequence and a preceding or succeeding frame of the inter-frame encoded video sequence on the basis of motion vectors therebetween along a first route between the particular frame and the preceding or succeeding frame;

ii) determine one or more further motion estimations representative of the global motion between the particular frame and the preceding or succeeding frame at least partially on the basis of motion vectors between the particular frame and one or more preceding or succeeding other frames along one or more second routes between the particular frame and the preceding or succeeding frame, wherein the one or more second routes are different from the first route; and

a motion estimation selector arranged in use to select one of the motion estimations which meets at least one predetermined criterion as being representative of the global motion of the frame.

14. A system according to claim 13, wherein the video processor is further arranged in use to:

determine one or more motion estimations representative of the global motion of the frame with respect to a one or more respective preceding or succeeding other frames;

determine one or motion estimations respectively representative of the global motion of the one or more other frames with respect to the preceding or succeeding frame; and

accumulate the respective motion estimations to give one or more respective overall motion estimations each substantially representative of the global motion of the frame with respect to the preceding or succeeding frame.

15. A system according to claim 13, wherein

the motion estimation selector further comprises a motion estimation tester for testing the motion estimations in turn, and providing an indication as to whether the test is passed;

and

an output unit arranged in use to output a motion estimation as being representative of the global motion of the frame in dependence on the receipt of an indication that the test was passed from the motion estimation tester, wherein

the motion estimation tester is further arranged to apply the test in turn to motion estimations once they have been determined, and if the test is passed then no further motion estimations are determined.

16. A system according to claim 15, wherein

the test comprises comparing the motion estimation with a threshold value, wherein the test is passed if the parameters of the motion estimation do not exceed the threshold value.

17. A system according to claim 15, and further comprising

a motion estimation interpolation unit arranged to interpolate between the motion estimations of adjacent frames to give an interpolated motion estimation, in dependence on the motion estimation testing means indicating that the test has been failed by all the motion estimations determined by the video processor.

18. A system for generating panoramic images from a motion-compensated inter-frame encoded video sequence, comprising:

a system for global motion estimation between frames of a motion-compensated inter-frame encoded video sequence, the system for global motion estimation comprising;

a video processor arranged in use to;

i) determine a motion estimation representative of the global motion directly between a particular frame of the inter-frame encoded video sequence and a preceding or succeeding frame of the inter-frame encoded video sequence on the basis of motion vectors therebetween along a first route between the particular frame and the preceding or succeeding frame;

ii) determine one or more further motion estimations representative of the global motion between the particular frame and the preceding or succeeding frame at least partially on the basis of motion vectors between sub-parts of the particular frame and sub-parts of one or more preceding or succeeding other frames along one or more second routes between the particular frame and the preceding or succeeding frame, wherein the one or more second routes are different from the first route; and

a motion estimation selector arranged in use to select one of the motion estimations which meets at least one predetermined criterion as being representative of the global motion of the frame;

and further arranged to provide global motion estimations for each frame; and

a panoramic image generating unit for generating at least one panoramic image representing the frames of the video sequence using the global motion estimations thus determined.

19. A system according to claim 18, wherein the panoramic image generating unit is further arranged in use to:

select a particular frame of the sequence as a reference frame, the plane of the reference frame thereby being a reference plane;

for each frame other than the reference frame, accumulate the global motion estimations from each frame back to the reference frame;

warp each frame other than the reference frame onto the reference plane using the accumulated global motion estimations to give one or more pixel values for each pixel in the reference plane; and

for each pixel position in the reference plane, select one of the available pixel values for use as the pixel value in the panoramic image.

20. A system according to claim 19, wherein

the panoramic image generating unit is further arranged to select a substantially median pixel value from the available pixel values for use in a background panoramic image.

21. A system according to claim 18, wherein

the panoramic image generating unit is further arranged to select a substantially most different pixel value from the available pixel values for use in a foreground panoramic image.

22. A system according to claim 18, wherein

the panoramic image generating unit is further arranged to:

calculate the mean pixel value of the available pixel values;

calculate the L1 distance between each available pixel value and the calculated mean pixel value; and

select the pixel value with the median L1 distance for use in a background panoramic image.

23. A system according to claim 18, wherein

the panoramic image generating unit is further arranged to:

calculate the mean pixel value of the available pixel values;

calculate the L1 distance between each available pixel value and the calculated mean pixel value; and

select the pixel value with the maximum L1 distance for use in a foreground panoramic image.