PROCESSING OF VIDEO DATA TO COMPENSATE FOR UNINTENDED CAMERA MOTION BETWEEN ACQUIRED IMAGE FRAMES

• US 20100208084A1
• Filed: 04/26/2010
• Published: 08/19/2010
• Est. Priority Date: 07/21/2004
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Zoran Corporation (Qualcomm, Inc.)
• Inventors: Pinto, Victor, Dvir, Itsik
• Granted Patent: US 8,289,401 B2
• Time in Patent Office: Days
• US Class Current: 348/207.100
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time
• CPC Class Codes: H04N 23/6842 by controlling the scanning position, e.g. windowing
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Video data that includes a plurality of images acquired over time are processed in an apparatus immediately after acquiring the video data, or during some later post-processing of the video data, to reduce the effects of unintended motion in the acquired images. Coarse and fine compensation for motion is performed on video frames that correspond to the acquired images. A coarse compensation value is employed to compress each video frame that is both coarse and fine motion compensated.

Claims

1. A method for enabling an apparatus to generate a video output, comprising:

employing a lens and a sensor to acquire video data that includes a plurality of images acquired over time, and performing actions on each acquired images, including;

determining a first global motion vector for a current image based on detected coarse motion between a plurality of earlier acquired images;

cropping an intermediate video frame from the current image, wherein the intermediate video frame is repositioned within a region of the current image to compensate for the detected coarse motion based on the determined first global motion vector;

determining a second global motion vector for the intermediate video frame based on detected fine motion between the intermediate video frame and at least one earlier intermediate video frame;

cropping a video frame from the intermediate video frame, wherein the video frame is repositioned within a region of the intermediate video frame to compensate for the detected fine motion based on the determined second global motion vector; and

compressing the video frame based on the determined first global motion vector; and

generating a video output that includes a plurality of the compressed video frames over time.

2. The method of claim 1, further comprising enabling a mechanical repositioning of the lens in an opposite direction of a detected movement of the apparatus to provide further coarse compensation for motion during acquisition of each image over time.

3. The method of claim 1, further comprising:

if a high rate of acquisition of video data is detected, compressing each intermediate video frame based on each correspondingly determined first global motion vector; and

generating another video output that includes a plurality of compressed intermediate video frames instead of the video output that includes the plurality of compressed video frames.

4. The method of claim 1, further comprising employing a low resolution version of earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.

5. The method of claim 1, further comprising:

if a low rate of acquisition of video data is detected, employing more than two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image; and

if a high rate of acquisition of video data is detected, employing two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.

6. An integrated circuit for enabling an apparatus to generate a video output, comprising:

a memory component that stores at least instructions; and

a processing component that executes instructions to enable a plurality of actions, including;

enabling a lens and a sensor to acquire video data that includes a plurality of images acquired over time, and performing actions on each acquired image, comprising;

determining a first global motion vector for a current image based on detected coarse motion between a plurality of earlier acquired images;

cropping an intermediate video frame from the current image, wherein the intermediate video frame is repositioned within a region of the current image to compensate for the detected coarse motion based on the determined first global motion vector;

determining a second global motion vector for the intermediate video frame based on detected fine motion between the intermediate video frame and at least one earlier intermediate video frame;

cropping a video frame from the intermediate video frame, wherein the video frame is repositioned within a region of the intermediate video frame to compensate for the detected fine motion based on the determined second global motion vector; and

compressing the video frame based on the determined first global motion vector; and

generating a video output that includes a plurality of the compressed video frames over time.

7. The integrated circuit of claim 6, wherein the actions further comprise enabling a mechanical repositioning of the lens in an opposite direction of a detected movement of the apparatus to provide further coarse compensation for motion during acquisition of each image over time.

8. The integrated circuit of claim 6, wherein the actions further comprise:

if a high rate of acquisition of video data is detected, compressing each intermediate video frame based on each correspondingly determined first global motion vector; and

generating another video output that includes a plurality of compressed intermediate video frames instead of the video output that includes the plurality of compressed video frames.

9. The integrated circuit of claim 6, wherein the actions further comprise employing a low resolution version of earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.

10. The integrated circuit of claim 6, wherein the actions further comprise:

if a low rate of acquisition of video data is detected, employing more than two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image; and

if a high rate of acquisition of video data is detected, employing two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.

11. A system for generating a video output, comprising:

a lens;

a sensor for acquiring an image projected by the lens onto the sensor;

a memory component for storing at least instructions; and

a processing component that executes the instructions to enable a plurality of actions, including;

enabling the lens and the sensor to acquire video data that includes a plurality of images acquired over time, and performing actions on each acquired image, comprising;

determining a first global motion vector for a current image based on detected coarse motion between a plurality of earlier acquired images;

cropping an intermediate video frame from the current image, wherein the intermediate video frame is repositioned within a region of the current image to compensate for the detected coarse motion based on the determined first global motion vector;

determining a second global motion vector for the intermediate video frame based on detected fine motion between the intermediate video frame and at least one earlier intermediate video frame;

cropping a video frame from the intermediate video frame, wherein the video frame is repositioned within a region of the intermediate video frame to compensate for the detected fine motion based on the determined second global motion vector; and

compressing the video frame based on the determined first global motion vector; and

generating the video output that includes a plurality of the compressed video frames over time.

12. The system of claim 11, wherein the actions further comprise enabling a mechanical repositioning of the lens in an opposite direction of a detected movement of the apparatus to provide further coarse compensation for motion during acquisition of each image over time.

13. The system of claim 11, wherein the actions further comprise:

if a high rate of acquisition of video data is detected, compressing each intermediate video frame based on each correspondingly determined first global motion vector; and

generating another video output that includes a plurality of compressed intermediate video frames instead of the video output that includes the plurality of compressed video frames.

14. The system of claim 11, wherein the actions further comprise employing a low resolution version of earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.

15. The system of claim 11, wherein the actions further comprise:

if a low rate of acquisition of video data is detected, employing more than two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image; and

if a high rate of acquisition of video data is detected, employing two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.

16. A processor readable storage media that is arranged to store executable instructions, wherein an execution of the stored instructions enables a plurality of actions for generating a video output that is motion compensated, the plurality of actions comprising:

enabling a lens and a sensor to acquire video data that includes a plurality of images acquired over time, and performing actions on each acquired image, comprising;

determining a first global motion vector for a current image based on detected coarse motion between a plurality of earlier acquired images;

cropping an intermediate video frame from the current image, wherein the intermediate video frame is repositioned within a region of the current image to compensate for the detected coarse motion based on the determined first global motion vector;

determining a second global motion vector for the intermediate video frame based on detected fine motion between the intermediate video frame and at least one earlier intermediate video frame;

cropping a video frame from the intermediate video frame, wherein the video frame is repositioned within a region of the intermediate video frame to compensate for the detected fine motion based on the determined second global motion vector; and

compressing the video frame based on the determined first global motion vector; and

generating a video output that includes a plurality of the compressed video frames over time.

17. The processor readable storage media of claim 16, wherein the actions further comprise enabling a mechanical repositioning of the lens in an opposite direction of a detected movement of the apparatus to provide further coarse compensation for motion during acquisition of each image over time.

18. The processor readable storage media of claim 16, wherein the actions further comprise:

if a high rate of acquisition of video data is detected, compressing each intermediate video frame based on each correspondingly determined first global motion vector; and

generating another video output that includes a plurality of compressed intermediate video frames instead of the video output that includes the plurality of compressed video frames.

19. The processor readable storage media of claim 16, wherein the actions further comprise a low resolution version of earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.

20. The processor readable storage media of claim 16, wherein the actions further comprise:

if a low rate of acquisition of video data is detected, employing more than two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image; and

if a high rate of acquisition of video data is detected, employing two earlier acquired images to detect coarse motion and determine the first global motion vector for the current image.