Compensating for undesirable camera shakes during video capture

• US 8,723,969 B2
• Filed: 03/20/2007
• Issued: 05/13/2014
• Est. Priority Date: 03/20/2007
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Garg, Shashank, Pore, Vinayak, Thakur, Ashish, Lin, Shang-Hung
• Primary Examiner(s): Moe, Aung S
• Assistant Examiner(s): Kao, Yih-Sien
• Application Number:
  US11/688,263
• Publication Number:
  US 20080231718A1
• Time in Patent Office: 2,611 Days
• Field of Search: 348/222.1, 348/208.99, 348/333.02, 348/208.1, 348/208.14
• US Class Current: 348/208.99
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

An image processor in an image capture device compensates for the effects of undesirable camera shakes occurring during video capture The image processor receives a pair of source frames representing images of a scene, generates a pair of subsampled frames from the source frames, and computes a coarse displacement of the captured image due to camera shakes by comparing the two subsampled frames. The image processor may then refine the determined coarse displacement by comparing the two source frames and a bound determined by an extent of subsampling, and compensate for the displacement accordingly. Display aberrations such as blank spaces caused due to shifting are also avoided by displaying only a portion of the captured image and shifting the displayed portion to compensate for camera shake. The image processor also recognizes displacements due to intentional camera movement, and does not correct for such displacements.

Claims

1. A method of obtaining stable video images of a scene using a video camera, said method comprising:

receiving a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances;

generating a first subsampled frame from a first set of pixels in said first source frame and a second subsampled frame from a second set of pixels in said second source frame;

computing a displacement of said second image relative to said first image;

refining said displacement by searching a plurality of pixel locations within said first and second sets of pixels in said first and second source frames for a correspondence to said first and second subsampled frames;

determining if said displacement is an intentional displacement by checking whether said sequence of displacements follows a uniform pattern; and

shifting said second source frame by an opposite of said displacement when said sequence of displacements does not follow a uniform pattern.

2. The method of claim 1, wherein said computing comprises:

calculating a second displacement by scaling said first displacement for said first and second source frames;

estimating a first displacement from said first subsampled frame and said second subsampled frame; and

calculating a second displacement by scaling said first displacement for said first and second source frames, said second displacement having a higher resolution than said first displacement.

3. The method of claim 2, wherein said higher resolution comprises a resolution equivalent to a resolution of the first and second source frames.

4. A method of obtaining stable video images of a scene using a video camera, said method comprising:

receiving a first set of pixel values representing a first image;

displaying a first area which covers only a first portion of said first image;

receiving a second set of pixel values representing a second image;

determining a displacement of said second image relative to said first image;

refining said displacement by searching a plurality of pixel locations comprised in said first area for a correspondence to said first portion;

determining if said displacement is an intentional displacement by checking whether said sequence of displacements follows a uniform pattern; and

displaying a second area which covers only a second portion of said second image, wherein said second area is shifted by opposite of said displacement in relation to said first area and said correspondence.

5. The method of claim 4, further comprising maintaining a window which specifies the portion of images displayed by said displaying, wherein said window is moved by opposite of said displacement prior to displaying of said second area.

6. The method of claim 5, wherein said window, said first image, said second image, said first area and said second area are all rectangles.

7. The method of claim 6, further comprising:

receiving a sequence of sets of pixel values following said second set of pixel values, said sequence of sets of pixel values representing a corresponding sequence of images, said sequence of sets of pixel values including a third set of pixel values and a fourth set of pixel values, wherein each displacement of said sequences of displacements is of an image relative to a preceding image in said sequence of images; and

shifting said window by said sequence of displacements in displaying corresponding portions of the images only if said sequence of displacements does not follow said uniform pattern.

8. A machine readable non-transitory medium carrying one or more sequences of instructions for obtaining stable video images of a scene using a video camera, wherein execution of said one or more sequences of instructions by one or more processors contained in said video camera causes said video camera to perform the actions of:

receiving a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances;

generating a first subsampled frame from a first set of pixels in said first source frame and a second subsampled frame from a second set of pixels in said second source frame;

computing a displacement of said second image relative to said first image;

refining said displacement by searching a plurality of pixel locations within said first and second sets of pixels in said first and second source frames for a correspondence to said first and second subsampled frames;

determining if said displacement is an intentional displacement by checking whether said sequence of displacements follows a uniform pattern; and

shifting said second source frame by an opposite of said displacement when said sequence of displacements does not follow a uniform pattern.

9. The machine readable non-transitory medium of claim 8, wherein said video camera estimates a first displacement based on image content represented by said first subsampled image and said second subsampled image, and determines said a second with a higher resolution than said first displacement by scaling said first displacement by comparing said first source image and said first subsampled frame.

10. The machine readable non-transitory medium of claim 9, wherein said higher resolution comprises a resolution equivalent to a resolution of the first and second source frames.

11. A machine readable non-transitory medium carrying one or more sequences of instructions for obtaining stable video images of a scene using a video camera, wherein execution of said one or more sequences of instructions by one or more processors contained in said video camera causes said video camera to perform the actions of:

receiving a first set of pixel values representing a first image;

displaying a first area which covers only a first portion of said first image;

receiving a second set of pixel values representing a second image;

determining a displacement of said second image relative to said first image, said displacement being comprised in a sequence of displacements corresponding to a plurality of images including said first and second images;

refining said displacement by searching a plurality of pixel locations within said first and set of pixel values in said first image for a correspondence to said first area;

determining if said displacement is an intentional displacement by checking whether said sequence of displacements follows a uniform pattern; and

displaying a second area which covers only a second portion of said second image, wherein said second area is shifted by opposite of said displacement in relation to said first area and said correspondence.

12. The machine readable non-transitory medium of claim 11, wherein said video camera maintains a window which specifies the portion of images displayed by said displaying, wherein said window is moved by opposite of said displacement prior to displaying of said second area.

13. The machine readable non-transitory medium of claim 12, wherein said window, said first image, said second image, said first area and said second area are all rectangles.

14. The machine readable non-transitory medium of claim 13, wherein said video camera receives a sequence of sets of pixel values following said second set of pixel values, said sequence of sets of pixel values representing a corresponding sequence of images, said sequence of sets of pixel values including a third set of pixel values and a fourth set of pixel values, wherein each displacement of said sequences of displacements is of an image relative to a preceding image in said sequence of images;

and

wherein said window is shifted by said sequence of displacements in displaying corresponding portions of the images only if said sequence of displacements does not follow said uniform pattern.

15. A video camera designed to provide stable video images of a scene, said video camera comprising:

an image sensor designed to receive light representing an image of said scene, and generating a plurality of frames including a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances; and

an image processor operable to;

receive said first source frame and said second source frame;

generate a first subsampled frame from a first set of pixels in said first source frame and a second subsampled frame from a second set of pixels in said second source frame;

compute a displacement of said second image relative to said first image by examining said first subsampled frame and said second subsampled frame;

refine said displacement by searching a plurality of pixel locations within said first and second sets of pixels in said first and second source frames for a correspondence to said first and second subsampled frames;

determine if said displacement is an intentional displacement by determining whether said displacement is comprised in a sequence of displacements following a uniform pattern; and

shift said second source frame by an opposite of said displacement when said sequence of displacements is determined to follow a uniform pattern.

16. The video camera of claim 15, wherein said image processor is further operable to:

estimate a first displacement based on image content represented by said first subsampled frame and said second subsampled frame; and

determine a second displacement with a higher resolution than said first displacement by scaling said first displacement by comparing said first source frame and said second source frame.

21. The video camera of claim 16, wherein said image process is further operable to search within an area formed by a number of pixels in said first and second source frames used to generate one pixel in the corresponding first and second subsampled frames.

17. A video camera designed to provide stable video images of a scene, said video camera comprising:

an image sensor designed to receive light representing an image of said scene, and generating a plurality of frames including a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances; and

an image processor operable to;

receive a first set of pixel values representing a first image;

display a first area which covers only a first portion of said first image;

receive a second set of pixel values representing a second image;

determine a displacement of said second set of pixel values relative to said first set of pixel values, said displacement being comprised in a sequence of displacements corresponding to a plurality of images including said first and second images;

refine said displacement by searching a plurality of pixel locations comprised in said first image for a correspondence in said first portion;

determine if said displacement is an intentional displacement by checking whether said sequence of displacements follows a uniform pattern; and

display a second area which covers only a second portion of said second image, wherein said second area is shifted by opposite of said displacement in relation to said first area.

18. The video camera of claim 17, wherein said image processor is operable to maintain a window which specifies the portion of images displayed by said displaying, wherein said window is moved by opposite of said displacement prior to displaying of said second area.

19. The video camera of claim 18, wherein said window, said first image, said second image, said first area and said second area are all rectangles.

20. The video camera of claim 19, wherein said image processor is further operable to:

receive a sequence of sets of pixel values following said second set of pixel values, said sequence of sets of pixel values representing a corresponding sequence of images, said sequence of sets of pixel values including a third set of pixel values and a fourth set of pixel values, wherein each displacement of said sequences of displacements is of an image relative to a preceding image in said sequence of images; and

shift said window by said sequence of displacements in displaying corresponding portions of the images only if said sequence of displacements does not follow said uniform pattern.