Processing of video data to compensate for unintended camera motion between acquired image frames

• US 7,705,884 B2
• Filed: 07/21/2004
• Issued: 04/27/2010
• Est. Priority Date: 07/21/2004
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Zoran Corporation (Qualcomm, Inc.)
• Inventors: Dvir, Itsik, Pinto, Victor
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): Adeyiga; Temitope
• Application Number:
  US10/897,186
• Publication Number:
  US 20060017814A1
• Time in Patent Office: 2,106 Days
• Field of Search: 348/208.1
• US Class Current: 348/208.99
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time
• CPC Class Codes: H04N 23/6842 by controlling the scanning position, e.g. windowing
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Video data of successive image frames are processed in a digital camera or other video image data acquisition device immediately after acquiring the video data, or during post-processing of the video data at some time after acquiring the video data, to reduce the effects of unintended motion (jitter) of the hand-held devices by stabilizing the images. A processing circuit used to calculate an estimate of motion between components of successive image frames as part of a MPEG-4 or other compression algorithm is also used to estimate motion upon which the video data are altered to stabilize the images. The individual images may be pre-stabilized by using the results of stabilizing a prior image in order to reduce the amount of processing necessary to stabilize a current image.

Claims

1. A method of operating a video device, comprising:

acquiring video data from a two-dimensional sensor of a succession of image frames;

pre-stabilizing individual image frames in sequence by individually cropping image frames of the acquired video data in accordance with motion vectors between images of two preceding image frames;

making a first calculation using a motion estimation algorithm based upon the acquired video data of a first set of quantities estimating motion of components of a current acquired image frame with respect to components of a another acquired image frame, said first calculation being made at least in part by passing the acquired video data though an electronic circuit that executes the given motion estimation algorithm;

calculating an estimate of an overall motion of the current image frame with respect to said another image frame from the first set of quantities estimating motion of components of the current frame with respect to components of said another frame;

modifying the acquired video data of the current image frame by the estimate of the overall motion of the current image frame with respect to said another image frame in a manner to provide data of a stabilized image frame;

compressing the data of the stabilized image frame, including passing the data of the stabilized image frame through said electronic circuit to make a second calculation of a second set of quantities estimating motion of components of the current stabilized image frames with respect to another stabilized image frame by executing said given motion estimation algorithm; and

storing the compressed data of the stabilized image frame.

2. The method of claim 1, wherein said another image frame includes an image frame within the succession of acquired image frames that is prior to the current image frame.

3. The method of claim 2, wherein the image frame that is prior to the current image frame immediately precedes the current image frame.

4. The method of claim 1, wherein making the calculation based upon the acquired video data by use of the given algorithm includes calculating estimates of motions of individual blocks within the current image frame with respect to motions of individual blocks within said another image frame.

5. The method of claim 4, wherein calculating an estimate of overall motion of the current image frame includes calculating a statistical value of the quantities representing motions of the individual blocks within the frame.

6. The method of claim 2, which additionally comprises, prior to making the calculation of quantities estimating motion of components of the current image frame, pre-stabilizing the current image frame as the result of a comparison of said prior image frame and an image frame acquired earlier than said prior image frame, and wherein making a calculation of quantities estimating motion of components of the current image frame does so with respect to components of said prior image frame after its pre-stabilization.

7. The method of claim 6, which additionally comprises, prior to pre-stabilizing the current image, of forming data of said prior image frame and said another image frame to have a resolution reduced from a resolution of the video data acquired from the sensor, and wherein pre-stabilizing the current image frame is performed by a comparison of said prior and said another image frames with reduced resolution.

8. The method of claim 7, wherein pre-stabilizing the current image frame includes defining the pre-stabilized current image frame to have a two-dimensional extent that lies within and which is shifted with respect to the current image frame of the acquired video data.

9. The method of claim 8, wherein modifying the current image frame includes altering coordinates defining the modified current image frame within the current image frame of the acquired video data.

10. The method of claim 1, wherein modifying the current image frame includes altering coordinates defining the modified current image frame within the acquired video data.

11. The method of claim 1, wherein calculating an estimate of the overall motion of the current image frame and modifying the current image frame by the estimate of overall motion are carried out by a processor executing controlling software.

12. The method of claim 11, wherein the calculations made by passing data through said electronic circuit are referred to the electronic circuit by the processor executing controlling software.

13. The method of claim 1, additionally comprising converting the acquired video data from a format representing the magnitude of two or more color components into a format representing the magnitude of luminance and chrominance components, and wherein making a calculation based upon the acquired video data includes making said calculation from data of the current and said prior image frames with data in the format representing the magnitude of luminance and chrominance components.

14. The method of claim 1, wherein each of the recited steps is carried out within a hand-held video image data acquisition device.

15. A method of processing video data of image frames acquired in succession of moving images, comprising:

acquiring video data of larger image frames than required for resulting processed image frames;

calculating a first set of global motion vectors representing estimates of global motion between individual pairs of image frames;

pre-stabilizing the individual image frames in sequence by individually cropping the image frames of the acquired video data in accordance with the motion vectors between images of two preceding image frames;

thereafter further stabilizing the individual pre-stabilized image frames in sequence by individually cropping the pre-stabilized image frames in accordance with the motion vectors between their images and images of immediately preceding frames; and

thereafter compressing data of the stabilized image frames.

16. The method of claim 15, additionally comprising:

calculating reduced resolution images of the individual image frames before pre-stabilizing the image frames, and

calculating a second set of global motion vectors representing estimates of global motion between individual pairs of reduced resolution image frames,

wherein pre-stabilizing the individual image frames includes pre-stabilizing the image frames in accordance with the second set of motion vectors.

20. The method of any one of claim 16, 18 or 19, wherein calculating the first and second sets of global motion vectors and compressing the data each include estimating a motion of components of image frames according to a common algorithm executed by an electronic circuit dedicated to such motion estimation.

21. The method of any one of claim 15 or 17, wherein each of the recited steps is carried out within a hand-held video image data acquisition device.

17. A method of processing video data of image frames acquired in succession of moving images, comprising:

acquiring video data of larger image frames than required for resulting processed image frames;

pre-stabilizing individual image frames in sequence by individually cropping image frames of the acquired video data in accordance with at least one motion vector between images of two preceding image frames;

calculating a first set of global motion vectors representing estimates of global motion between individual pairs of image frames;

stabilizing the individual image frames in sequence by individually cropping the image frames of the acquired video data in accordance with the motion vectors between images of two others of the acquired image frames; and

thereafter compressing data of the stabilized image frames.

18. The method of claim 17, additionally comprising:

calculating reduced resolution images of the individual image frames before stabilizing the image frames, and

calculating a second set of global motion vectors representing, estimates of global motion between individual pairs of reduced resolution image frames,

wherein stabilizing the individual image frames includes stabilizing the image frames in accordance with the second set of motion vectors.

19. The method of claim 17, additionally comprising:

calculating reduced resolution images of the individual image frames before stabilizing the image frames, and

calculating a second set of global motion vectors representing estimates of global motion between individual pairs of reduced resolution image frames,

wherein stabilizing the individual image frames is performed by stabilizing the image frames in accordance with the second set of motion vectors.

22. A method of operating a video device, comprising:

acquiring video data from a two-dimensional sensor of a succession of image frames;

pre-stabilizing individual image frames in sequence by individually cropping image frames of the acquired video data in accordance with at least one motion vector between images of two preceding image frames;

making first calculations using a motion estimation algorithm based upon the video data of a first set of quantities that estimate motion of components of individual ones of the image frames in process with respect to components of one or more other ones of the image frames;

calculating estimates of overall motion of the image frames from the first set of quantities that estimate motion of components of the individual image frames;

modifying the acquired video data by the estimates of overall motion of the image frames in a manner to provide data of stabilized image frames;

making second calculations using the motion estimation algorithm based upon the stabilized image frames of a second set of quantities that estimate motion of components of individual ones of the stabilized image frames in process with respect to components of one or more other ones of the stabilized image frames;

utilizing the second set of quantities that estimate motion of components of the stabilized image frames to compress data of the stabilized image frames; and

storing the compressed data of the stabilized image frames.

23. The method of claim 22, wherein making calculations of quantities that estimate motion of components of the individual image frames includes passing the acquired video data through an electronic circuit that executes a given motion estimation algorithm.

24. The method of claim 22, wherein calculating estimates of overall motion of the individual image frames includes doing so from quantities that estimate motion of components of image frames occurring prior to the individual frame for which the motion estimates are being calculated.

25. The method of claim 22, wherein calculating estimates of overall motion of the individual image frames includes doing so from quantities that estimate motion of components of image frames occurring subsequent to the individual frame for which the motion estimates are being calculated.

26. The method of claim 22, wherein calculating estimates of overall motion of the individual image frames includes doing so by omitting use of quantities that estimate motion of the individual image frame for which the motion estimates are being calculated.

27. The method of any one of claim 22, 23 or 26, wherein the one or more other image frames occur before the individual image frame for which the motion estimates are being calculated.

28. The method of any one of claim 22, 23 or 26, wherein the one or more other image frames occur after the individual image frame for which the motion estimates are being calculated.

29. The method of any one of claims 22-26, wherein acquiring video data includes acquiring the succession of image frames at a rate of at least thirty frames-per-second.

30. A method of processing video data of frames of moving images that occur in sequence, which comprises:

pre-stabilizing individual image frames in sequence by individually cropping image frames of the video data in accordance with at least one motion vector between images of two preceding image frames;

calculating a first set of motion vectors representing estimates of global motion of a first image frame by use of data of the first image frame and data of another image frame;

compressing the data of the first image frame by use of the first set of motion vectors; and

stabilizing the data of a second image frame following the first image frame by use of the first set of motion vectors.

31. An integrated circuit chip, comprising:

a first set of conductors on the chip that carry digital data of a succession of video image frames,

an electronic circuit that calculates motion vectors of image frames whose digital data are passed there through, using a motion estimation algorithm,

a data processor programmed to cause digital data of image frames received through the first set of conductors to pass through the electronic circuit, thereby to pre-stabilize individual image frames in sequence by individually cropping image frames of the received digital data in accordance with motion vectors between images of two preceding image frames, to thereafter calculate a first set of motion vectors on the received image frame data using the motion estimation algorithm, to thereafter cause the image frames to be stabilized by use of the first set of motion vectors calculated from the received data motion vectors, to thereafter cause data of the stabilized image frames to be passed through the electronic circuit, thereby to calculate a second set of motion vectors on the stabilized image frames using the motion estimation algorithm, and to thereafter compress the stabilized image frames by use of the calculated second set of motion vectors on the stabilized image frames, and

a second set of conductors on the chip that carry data of the compressed stabilized image frames.

32. A video system, comprising:

an input of data of successive frames of a moving image,

a processor configured and arranged to perform the following instructions;

pre-stabilizing individual frames in sequence by individually cropping frames of the inputted data in accordance with at least one motion vector between images of two preceding frames;

making a first calculation using a motion estimation algorithm based upon a current frame of the successive frames of a first set of quantities estimating motion of components of the current frame with respect to components of a another frame of the successive frames, said first calculation being made at least in part by passing the acquired video data through an electronic circuit that executes the given motion estimation algorithm,

calculating an estimate of an overall motion of the current frame with respect to said another frame from the first set of quantities estimating motion of components of the current frame with respect to components of said another frame,

modifying the acquired video data of the current frame by the estimate of the overall motion of the current frame with respect to said another frame in a manner to provide data of a stabilized frame, and

compressing the data of the stabilized frame, make a second calculation of a second set of quantities estimating motion of components of the stabilized frame with respect to another stabilized frame by executing said given motion estimation algorithm, and

an output of data of successive stabilized and compressed versions of the input data of the successive frames.