System and method for video stabilization

• US 20070076982A1
• Filed: 09/30/2005
• Published: 04/05/2007
• Est. Priority Date: 09/30/2005
• Status: Abandoned Application

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Petrescu, Doina
• Application Number:
  US11/241,666
• Publication Number:
  US 20070076982A1
• Time in Patent Office: Days
• US Class Current: 382/294
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6842 by controlling the scanning position, e.g. windowing
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20

Abstract

Disclosed is a method and circuit for stabilizing unintentional motion within an image sequence generated by an image capturing device (102). The image sequence is formed from a temporal sequence of frames, each frame (202) having an area and an outer boundary. The images are two dimensional arrays of pixels. The area of the frames is divided into a foreground area portion (204) and background area portion (206). From the background area portion of the frames, a background pixel domain is selected for evaluation (404). The background pixel domain is used to generate an evaluation (406), for subsequent stabilization processing (408), calculated between corresponding pairs of a sub-sequence of select frames.

Claims

1. A method for stabilizing elements within an image sequence formed from a temporal sequence of frames, each frame having an area, the image sequence generated by an image capturing device, the method comprising:

dividing the area of the frames of the sequence of frames into sub-areas comprising a foreground area portion and background area portion;

selecting a background pixel domain for evaluation from the background area portion of the frames;

evaluating the background pixel domain to generate an evaluation for subsequent stabilization processing calculated between corresponding pairs of a sub-sequence of select frames; and

applying stabilization processing based on the evaluation to the frames of the sequence of frames.

2. A method as recited in claim 1 wherein prior to applying the stabilization processing, the frames comprise an outer boundary from which a buffer region is formed, wherein the buffer region is used during the stabilization processing to supply image information including spare row data and column data.

3. A method as recited in claim 1 wherein the sub-sequence of select frames comprises consecutive select frames.

4. A method as recited in claim 1 wherein selecting the background pixel domain from the background area portion in the frames, comprises:

determining corner sectors of the frames of the sequence of frames; and

forming the background pixel domain to correspond to the corner sectors.

5. A method as recited in claim 1 wherein selecting the background pixel domain from the background area portion in the frames comprises:

determining a center sector substantially corresponding to the foreground area portion; and

forming the background pixel domain to substantially correspond to an area portion in the frames of the sequence of frames outside the center sector.

6. A method as recited in claim 1 wherein selecting further comprises selecting a plurality of background pixel domains from the background area portion in the frames of the sequence of frames, the method comprising:

selecting a predetermined number of background pixel domains.

7. A method as recited in claim 1 wherein selecting further comprises selecting a plurality of background pixel domains from the background area portion in the frames of the sequence of frames, the method comprising:

selecting four background pixel domains.

8. A method as recited in claim 1 wherein a background pixel domain comprises select pixel groupings, and wherein evaluating the background pixel domain for subsequent stabilization processing, comprises:

calculating displacement components of elements within the pixel groupings to generate the evaluation.

9. A method as recited in claim 8 wherein the displacement components include a pair of substantially orthogonal displacement vectors.

10. A method as recited in claim 8 wherein the pixel arrays comprise pixel values, and wherein calculating displacement components comprises:

summing the pixel values in a vertical direction to determine a horizontal displacement vector; and

summing the pixel values in a horizontal direction to determine a vertical displacement vector.

11. A method as recited in claim 10 wherein applying stabilization processing based on the evaluation, comprises:

calculating a global motion vector by determining an average of middle range values for the vertical displacements components and an average of middle range values for the horizontal displacement components.

12. A method as recited in claim 1 wherein dividing the area of the frames of the sequence of frames into sub-areas comprising a foreground area portion and background area portion is performed manually.

13. A method as recited in claim 1 wherein dividing the area of frames of a sequence of frames into sub-areas comprising a foreground area portion and background area portion, comprises:

determining the background area portion by locating a sub-area comprising a motion amplitude value that is below a predetermined threshold value.

14. A method as recited in claim 1 wherein selecting the background pixel domain comprises;

locating one or more sub-areas that are substantially uniformly static between evaluated frames.

15. A method as recited in claim 1 wherein dividing the area of frames of a sequence of frames into sub-areas comprising a foreground area portion and background area portion, comprises:

determining the foreground area portion by locating a sub-area having motion.

16. A method as recited in claim 1, comprising:

processing the dividing, selecting, evaluating and applying steps while the frames in the image sequence formed from the temporal sequence are being generated by the image capturing device.

17. A method for stabilizing elements within an image sequence formed from a temporal sequence of frames, each frame having an area, the image sequence generated by an image capturing device, the method comprising:

determining boundary regions of the frames of the sequence of frames;

selecting the boundary regions for evaluation of the frames;

evaluating the corresponding selected boundary regions to generate an evaluation for subsequent stabilization processing calculated between corresponding pairs of a sub-sequence of select frames; and

applying stabilization processing based on the evaluation to the frames of the sequence of frames.

18. A method as recited in claim 17, wherein the selected boundary regions comprise one or more corner sectors.

19. A method as recited in claim 17, wherein the selected boundary region is substantially comprised of background area portions.

20. A method as recited in claim 18 wherein the corner sectors comprise pixels arrayed orthogonally to form pixel arrays, and wherein evaluating the selected boundary regions for subsequent stabilization processing, comprises:

calculating displacements components of select pixel groupings within the selected boundary regions to generate the evaluation.

21. A method as recited in claim 20 wherein the pixels comprise pixel values, and wherein calculating displacement components comprises:

summing the pixel values in a vertical direction to determine horizontal displacement components; and

summing the pixel values in a horizontal direction to determine vertical displacement components.

22. A method as recited in claim 21 wherein evaluating the vertical displacements components and the horizontal displacement components, comprises:

evaluating the vertical displacement components and the horizontal displacement components separately.

23. A circuit for stabilizing an image sequence formed from a sequence of frames, each frame having an area, the image sequence generated by an image capturing device, the method comprising:

a determining module for determining corner sectors of the area of the frames of the sequence of frames;

a forming module for forming a background pixel domain to correspond to the corner sectors;

an evaluation module for evaluating the background pixel domain to generate an evaluation for subsequent stabilization processing; and

an application module for applying stabilization processing based on the evaluation to the area of the frames of the sequence of frames.

24. A system as recited in claim 23 wherein the background pixel domain comprises vertical pixel columns and horizontal pixel rows, and wherein the evaluation module comprises:

a determination module for determining vertical displacements components of the vertical pixel columns and the horizontal displacement components of the horizontal pixel rows of the frames of the sequence of frames to generate the evaluation.

25. A system as recited in claim 23 wherein the evaluation module comprises:

separate evaluation modules for evaluating the vertical displacement components and the horizontal displacement components separately.

26. A system as recited in claim 25 further comprising:

a calculation module calculating a global motion vector by determining an average of middle range values for the vertical displacements components and an average of middle range values for the horizontal displacement components.