Adaptive image stabilization

• US 7,880,769 B2
• Filed: 02/13/2004
• Issued: 02/01/2011
• Est. Priority Date: 02/13/2004
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Qi, Yingyong
• Primary Examiner(s): Tran; Sinh
• Assistant Examiner(s): Daniels; Anthony J
• Application Number:
  US10/779,246
• Publication Number:
  US 20050179784A1
• Time in Patent Office: 2,545 Days
• Field of Search: 34820899-2086
• US Class Current: 348/208.99
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method and apparatus for image stabilization takes an input image sequence including a plurality of frames, estimates frame-level motion vectors for each frame, and adaptively integrates the motion vectors to produce, for each frame, a motion vector to be used for image stabilization. A copy of the reference image of a frame is displaced by the corresponding adaptively integrated motion vector. In one embodiment, the perimeter of the image sensor is padded with a margin to be used for image compensation. In another embodiment, vertical and horizontal components are treated independently. In still another embodiment, the motion estimation circuitry associated with an MPEG-4 encoder is used to calculate macroblock level vectors, and a histogram is used to compute a corresponding frame-level vector for that frame.

Claims

1. A method of stabilizing an image comprising a plurality of sequential frames, comprising:

estimating a plurality of motion vectors, each motion vector corresponding to one of the frames;

adaptively integrating each of the motion vectors with the motion vectors corresponding to the previous frames using a stabilizer circuit; and

using the adaptively integrated motion vectors to compensate the frames to stabilize the image,

wherein the adaptive integration comprises setting one of the adaptively integrated motion vectors equal to the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame when the product of said one of adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous flame is less than zero.

2. A method of stabilizing an image comprising a plurality of sequential frames, comprising:

estimating a plurality of motion vectors, each motion vector corresponding to one of the frames;

adaptively integrating each of the motion vectors with the motion vectors corresponding to the previous frames using a stabilizer circuit; and

using the adaptively integrated motion vectors to compensate the frames to stabilize the image,

wherein the adaptive integration comprises setting one of the adaptively integrated motion vectors equal to a margin having the sign of the adaptively integrated motion vector corresponding to the previous frame when the absolute value of the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is greater than the margin.

3. A method of stabilizing an image comprising a plurality of sequential frames, comprising:

estimating a plurality of motion vectors, each motion vector corresponding to one of the frames;

adaptively integrating each of the motion vectors with the motion vectors corresponding to the previous frames using a stabilizer circuit; and

using the adaptively integrated motion vectors to compensate the frames to stabilize the image,

wherein the adaptive integration comprises setting each of the adaptively integrated motion vectors F(n) equal to

[(1−

V(n)/M)×

F(n−

1)]+V(n)

when |V(n)+F(n−

1)|<

M, where

V(n) equals one of the motion vectors, M equals the margin, and F(n−

1) equals the adaptively integrated motion vector corresponding to the previous frame.

4. A method for image stabilization of an image sequence comprising n frames, comprising:

storing each frame from an image sensor into a reference buffer;

estimating a frame-level motion vector V(n) for each of the n frames;

selecting an integration approach for each frame-level motion vector based upon components associated with the frame-level motion vector;

generating adaptively, according to the selecting, integrated motion vectors F(n) for each of the n frames based on V(n) and a motion vector F(n−

1) corresponding to the previous n−

1 frames using a stabilizer circuit, wherein the generating further comprises at least one of;

setting one of the adaptively integrated motion vectors equal to the sum of one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to a previous frame when a product of said one of adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is less than zero,

setting one of the adaptively integrated motion vectors equal to a margin having a sign of the adaptively integrated motion vector corresponding to the previous frame when an absolute value of the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is greater than the margin, or

setting each of the adaptively integrated motion vectors F(n) equal to

[(1−

V(n)/M)×

F(n−

1)]+V(n)

when |v(n)+F(n−

1)|<

M, where

M equals the margin; and

rendering each of the frames as a video sequence by displacing, when necessary, the nth frame in the reference buffer by F(n).

5. The method of claim 4, wherein the estimating the frame-level motion vector V(n) comprises:

generating, for each nth frame, a plurality of motion vectors, each motion vector corresponding to a specific area of the frame;

generating vertical and horizontal components of a histogram of the plurality of motion vectors corresponding to each nth frame; and

computing the frame-level vector V(n) based on the histogram.

6. The method of claim 4 further comprising padding the perimeter of the image sensor with a margin.

7. A method for image stabalization of an image sequence comprising n frames, comprising:

storing each frame from an image sensor into a reference buffer;

estimating a frame-level motion vector V(n) for each of the n frames;

generating adaptively integrated motion vectors F(n) for each of the n frames based on V(n) and a motion vector F(n−

I) corresponding to the previous n−

1 frames using a stabalizer circuit; and

rendering each of the n frames as a video sequence by displacing, when necessary, the nth frame in the reference buffer by F(n),

wherein the generating adaptively integrated motion vectors F(n) comprises setting the following values;

F(n)=F(n−

1) when V(n)=0;

F(n)=V(n)+F(n−

1) when V(n)×

F(n−

1)<

0;

F(n)=sgn(V(n))×

M, when |V(n)+F(n−

1)|≧

M, where M=margin;

F(n)=[(1−

V(n)/M)=F(n−

1)]+V(n) when |V(n)+F(n−

1)|<

M.

8. The method of claim 7, wherein the horizontal and vertical directions of each vector F(n), V(n), and F(n+1) are computed separately.

9. An image stabilization apparatus for use in a device capable of capturing video, comprising:

an image sensor for capturing video frames comprising an image sequence;

a buffer coupled to the image sensor for storing a reference image associated with a captured frame;

a motion estimation circuit coupled to the buffer and operative to compute a motion vector for each frame;

an adaptive integration circuit coupled to the motion estimation circuit and operative to adaptively integrate the motion vectors associated with each frame to output a cumulative motion vector for each frame; and

a rendering circuit coupled to the buffer and the adaptive integration circuit and operative to copy a stabilized portion of the image sequence,

wherein the adaptive integration circuit is further operative to compute F(n)=V(n)+F(n−

1) when V(n)=F(n)<

0.

10. An image stabilization apparatus for use in a device capable of capturing video, comprising:

an image sensor for capturing video frames comprising an image sequence;

a buffer coupled to the image sensor for storing a reference image associated with a captured frame;

a motion estimation circuit coupled to the buffer and operative to compute a motion vector for each frame;

an adaptive integration circuit coupled to the motion estimation circuit and operative to adaptively integrate the motion vectors associated with each frame to output a cumulative motion vector for each frame; and

a rendering circuit coupled to the buffer and the adaptive integration circuit and operative to copy a stabilized portion of the image sequence,

wherein the adaptive integration circuit is further operative to compute F(n)=sgn(V(n))×

M when |V(n)+F(n−

1)|≧

M.

11. An image stabilization apparatus for use in a device capable of capturing video, comprising:

an image sensor for capturing video frames comprising an image sequence;

a buffer coupled to the image sensor for storing a reference image associated with a captured frame;

a motion estimation circuit coupled to the buffer and operative to compute a motion vector for each frame;

an adaptive integration circuit coupled to the motion estimation circuit and operative to adaptively integrate the motion vectors associated with each frame to output a cumulative motion vector for each frame; and

a rendering circuit coupled to the buffer and the adaptive integration circuit and operative to copy a stabilized portion of the image sequence,

wherein the adaptive integration circuit is further operative to compute F(n)=(1−

V(n)/M)F(n−

1)+V(n) when |V(n)+F(n−

1)<

M.

12. An apparatus for stabilizing an image comprising a plurality of sequential frames, comprising:

means for estimating a plurality of motion vectors, each motion vector corresponding to one of the frames;

means for adaptively integrating each of the motion vectors with the motion vectors corresponding to the previous frames; and

means for using the adaptively integrated motion vectors to compensate the frames to stabilize the image,

wherein the adaptive integration comprises means for setting one of the adaptively integrated motion vectors equal to the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame when the product of said one of adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is less than zero.

13. An apparatus for stabilizing an image comprising a plurality of sequential frames, comprising:

means for estimating a plurality of motion vectors, each motion vector corresponding to one of the frames;

means for adaptively integrating each of the motion vectors with the motion vectors corresponding to the previous frames; and

means for using the adaptively integrated motion vectors to compensate the frames to stabilize the image,

wherein the adaptive integration comprises means for setting one of the adaptively integrated motion vectors equal to a margin having the sign of the adaptively integrated motion vector corresponding to the previous frame when the absolute value of the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is greater than the margin.

14. An apparatus for stabilizing an image comprising a plurality of sequential frames, comprising:

means for estimating a plurality of motion vectors, each motion vector corresponding to one of the frames;

means for adaptively integrating each of the motion vectors with the motion vectors corresponding to the previous frames; and

means for using the adaptively integrated motion vectors to compensate the frames to stabilize the image,

wherein the adaptive integration comprises means for setting each of the adaptively integrated motion vectors F(n) equal to

[(1−

V(n)/M)=F(n−

1)]+V+(n)

when |v(n)+F(−

1)|<

M, where

V(n) equals one of the motion vectors, M equals the margin, and F(n−

1) equals the adaptively integrated motion vector corresponding to the previous frame.

15. An apparatus for image stabilization of an image sequence comprising n frames, comprising:

means for storing each frame from an image sensor into a reference buffer;

means for estimating a frame-level motion vector V(n) for each of the n frames;

means for selecting an integration approach for each frame-level motion vector based upon components associated with the frame-level motion vector;

means for generating adaptively, according to the means for selecting, integrated motion vectors F(n) for each of the n frames based on V(n) and a motion vector F(n−

1) corresponding to the previous n−

1 frames, wherein the means for generating further comprises at least one of;

means for setting one of the adaptively integrated motion vectors equal to the sum of one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to a previous frame when a product of said one of adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is less than zero,

means for setting one of the adaptively integrated motion vectors equal to a margin having a sign of the adaptively integrated motion vector corresponding to the previous frame when an absolute value of the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is greater than the margin, or

means for setting each of the adaptively integrated motion vectors F(n) equal to

[(1−

V(n)/M)=F(n−

1)]+V(n)

when |V(n)+F(n−

1)|<

M, where

M equals the margin; and

means for rendering each of the n frames as a video sequence by displacing, when necessary, the nth frame in the reference buffer by F(n).

16. The apparatus of claim 15, wherein the means for estimating the frame-level motion vector V(n) comprises:

means for generating, for each nth frame, a plurality of motion vectors, each motion vector corresponding to a specific area of the frame;

means for generating vertical and horizontal components of a histogram of the plurality of motion vectors corresponding to each nth frame; and

means for computing the frame-level vector V(n) based on the histogram.

17. The apparatus of claim 15, further comprising means for padding the perimeter of the image sensor with a margin.

18. An apparatus for image stabilization of an image sequence comprising n frames, comprising:

means for storing each frame from an image sensor into a reference buffer;

means for estimating a frame-level motion vector V(n) for each of the n frames;

means for generating adaptively integrated motion vectors F(n) for each of the n frames based on V(n) and a motion vector F(n−

1) corresponding to the previous n−

1 frames; and

means for rendering each of the n frames as a video sequence by displacing, when necessary, the nth frame in the reference buffer by F(n),

wherein the means for generating adaptively integrated motion vectors F(n) comprises means for setting the following values;

F(n)=F(−

1) when V(n)=0;

F(n)=V(n)+F(n−

1) when V(n)×

F(n−

1<

0;

F(n)=sgn(V(n))×

M, when |V(n)+F(n−

1)|≧

M, where M=margin;

F(n)=[(1−

V(n)/M)×

F(n−

1)]+V(n) when |V(n)+F(n−

1)<

M.

19. The apparatus of claim 18, wherein the horizontal and vertical directions of each vector F(n), V(n), and F(n+1) are computed separately.

20. A non-transitory computer readable medium having program code executable by a processor to perform image stabilization of an image comprising a plurality of sequential frames, comprising:

program code to estimate a plurality of motion vectors, each motion vector corresponding to one of the frames;

program code to adaptively integrate each of the motion vectors with the motion vectors corresponding to the previous frames; and

program code to use the adaptively integrated motion vectors to compensate the frames to stabilize the image, wherein the adaptive integration comprises program code to set one of the adaptively integrated motion vectors equal to the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame when the product of said one of adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is less than zero.

21. A non-transitory computer readable medium having program code executable by a processor to perform image stabilization of an image comprising a plurality of sequential frames, comprising:

program code to estimate a plurality of motion vectors, each motion vector corresponding to one of the frames;

program code to adaptively integrate each of the motion vectors with the motion vectors corresponding to the previous frames; and

program code to use the adaptively integrated motion vectors to compensate the frames to stabilize the image, wherein the adaptive integration comprises program code to set one of the adaptively integrated motion vectors equal to a margin having the sign of the adaptively integrated motion vector corresponding to the previous frame when the absolute value of the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is greater than the margin.

22. A non-transitory computer readable medium having program code executable by a processor to perform image stabilization of an image comprising a plurality of sequential frames, comprising:

program code to estimate a plurality of motion vectors, each motion vector corresponding to one of the frames;

program code to adaptively integrate each of the motion vectors with the motion vectors corresponding to the previous frames; and

program code to use the adaptively integrated motion vectors to compensate the frames to stabilize the image,

wherein the adaptive integration comprises program code to set each of the adaptively integrated motion vectors F(n) equal to

[(1−

V(n)/M)×

F(n−

1)]+V(n)

when|v(n)+F(n−

1)|<

M, where

V(n) equals one of the motion vectors, M equals the margin, and F(n−

1) equals the adaptively integrated motion vector corresponding to the previous frame.

23. A non-transitory computer readable medium having program code executable by a processor to perform image stabilization of an image sequence comprising n frames, comprising:

program code to store each frame from an image sensor into a reference buffer;

program code to estimate a frame-level motion vector V(n) for each of the n frames;

program code to select an integration approach for each frame-level motion vector based upon components associated with the frame-level motion vector;

program code to generate adaptively, according to the program code to select, integrated motion vectors F(n) for each of the n frames based on V(n) and a motion vector F(n−

1) corresponding to the previous n−

1 frames, wherein the program code to generate further comprises at least one of;

program code to set one of the adaptively integrated motion vectors equal to the sum of one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to a previous frame when a product of said one of adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is less than zero,

program code to set one of the adaptively integrated motion vectors equal to a margin having a sign of the adaptively integrated motion vector corresponding to the previous frame when an absolute value of the sum of said one of the adaptively integrated motion vectors and the adaptively integrated motion vector corresponding to the previous frame is greater than the margin, or

program code to set each of the adaptively integrated motion vectors F(n) equal to

[(1−

V(n)/M)×

F(n−

1)] +V(n)

when |V(n)+F(n−

1)|<

M, where

M equals the margin; and

program code to render each of the n frames as a video sequence by displacing, when necessary, the nth frame in the reference buffer by F(n).

24. The non-transitory computer readable medium of claim 23, wherein the program code to estimate the frame-level motion vector V(n) comprises:

program code to generate, for each nth frame, a plurality of motion vectors, each motion vector corresponding to a specific area of the frame;

program code to generate vertical and horizontal components of a histogram of the plurality of motion vectors corresponding to each nth frame; and

program code to compute the frame-level vector V(n) based on the histogram.

25. The non-transitory computer readable medium of claim 23 further comprising program code to pad the perimeter of the image sensor with a margin.

26. A non-transitory computer readable medium having program code executable by a processor to perform image stabilization of an image sequence comprising n frames, comprising:

program code to store each frame from an image sensor into a reference buffer;

program code to estimate a frame-level motion vector V(n) for each of the n frames;

program code to generate adaptively integrated motion vectors F(n) for each of the n frames based on V(n) and a motion vector F(n−

1) corresponding to the previous n−

1frames; and

program code to render each of the n frames as a video sequence by program code to displac, when necessary, the nth frame in the reference buffer by F(n),

wherein the program code to generate adaptively integrated motion vectors F(n) comprises program code to set the following values;

F(n)=F(n−

1) when V(n)=0;

F(n)=V(n)+F(n−

1) when V(n) x F(n−

1)<

0;

F(n)=sgn(V(n))×

M, when |V(n)+F(n−

1)|≧

M, where M=margin;

F(n)=[(1−

V(n)/M)×

F(n−

1)]+V(n) when|V(n+F(n−

1)|<

M.

27. The non-transitory computer readable medium of claim 26, wherein the horizontal and vertical directions of each vector F(n), V(n), and F(n+1) are computed separately.