Compensating for Undesirable Camera Shakes During Video Capture

• US 20080231718A1
• Filed: 03/20/2007
• Published: 09/25/2008
• Est. Priority Date: 03/20/2007
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Pore, Vinayak, Garg, Shashank, Lin, Shang-Hung, Thakur, Ashish
• Granted Patent: US 8,723,969 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

An image processor in an image capture device compensates for the effects of undesirable camera shakes occurring during video capture The image processor receives a pair of source frames representing images of a scene, generates a pair of subsampled frames from the source frames, and computes a coarse displacement of the captured image due to camera shakes by comparing the two subsampled frames. The image processor may then refine the determined coarse displacement by comparing the two source frames and a bound determined by an extent of subsampling, and compensate for the displacement accordingly. Display aberrations such as blank spaces caused due to shifting are also avoided by displaying only a portion of the captured image and shifting the displayed portion to compensate for camera shake. The image processor also recognizes displacements due to intentional camera movement, and does not correct for such displacements.

Claims

1. A method of obtaining stable video images of a scene using a video camera, said method comprising:

receiving a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances;

generating a first subsampled frame from said first source frame, and a second subsampled frame from said second source frame;

computing a displacement of said second image relative to said first image by examining said first subsampled frame and said second subsampled frame; and

shifting said second source frame by opposite of said displacement.

2. The method of claim 1, wherein said computing comprises:

estimating a coarse displacement based on image content represented by said first subsampled image and said second subsampled image; and

determining said displacement with more accuracy than said coarse displacement by comparing said first source image and said second source image.

3. The method of claim 2, wherein said determining searches within an area formed by a number of pixels in said source frames used to generate one pixel in the corresponding subsampled frames.

4. A method of obtaining stable video images of a scene using a video camera, said method comprising:

receiving a first set of pixel values representing a first image;

displaying a first area which covers only a first portion of said first image;

receiving a second set of pixel values representing a second image;

determining a displacement of said second image relative to said first image; and

displaying a second area which covers only a second portion of said second image, wherein said second area is shifted by opposite of said displacement in relation to said first area.

5. The method of claim 4, further comprising maintaining a window which specifies the portion of images displayed by said displaying, wherein said window is moved by opposite of said displacement prior to displaying of said second area.

6. The method of claim 5, wherein said window, said first image, said second image, said first area and said second area are all rectangles.

7. The method of claim 6, further comprising:

receiving a sequence of sets of pixel values following said second set of pixel values, said sequence of sets of pixel values representing a corresponding sequence of images, said sequence of sets of pixel values including a third set of pixel values and a fourth set of pixel values;

determining a sequence of displacements, wherein each displacement is of image relative to a preceding image in said sequence of images;

checking whether said sequence of displacements follow a uniform pattern; and

shifting said window by said sequence of displacements in displaying corresponding portions of the images only if said sequence of displacements do not follow said uniform pattern.

8. A machine readable medium carrying one or more sequences of instructions for obtaining stable video images of a scene using a video camera, wherein execution of said one or more sequences of instructions by one or more processors contained in said video camera causes said video camera to perform the actions of:

receiving a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances;

generating a first subsampled frame from said first source frame, and a second subsampled frame from said second source frame;

computing a displacement of said second image relative to said first image by examining said first subsampled frame and said second subsampled frame; and

shifting said second source frame by opposite of said displacement.

9. The machine readable medium of claim 8, wherein said video camera estimates a coarse displacement based on image content represented by said first subsampled image and said second subsampled image, and determines said displacement with more accuracy than said coarse displacement by comparing said first source image and said second source image.

10. The machine readable medium of claim 9, wherein said determining searches within an area formed by a number of pixels in said source frames used to generate one pixel in the corresponding subsampled frames.

11. A machine readable medium carrying one or more sequences of instructions for obtaining stable video images of a scene using a video camera, wherein execution of said one or more sequences of instructions by one or more processors contained in said video camera causes said video camera to perform the actions of:

receiving a first set of pixel values representing a first image;

displaying a first area which covers only a first portion of said first image;

receiving a second set of pixel values representing a second image;

determining a displacement of said second image relative to said first image; and

displaying a second area which covers only a second portion of said second image, wherein said second area is shifted by opposite of said displacement in relation to said first area.

12. The machine readable medium of claim 11, wherein said video camera maintains a window which specifies the portion of images displayed by said displaying, wherein said window is moved by opposite of said displacement prior to displaying of said second area.

13. The machine readable medium of claim 12, wherein said window, said first image, said second image, said first area and said second area are all rectangles.

14. The machine readable medium of claim 13, wherein said video camera

receives a sequence of sets of pixel values following said second set of pixel values, said sequence of sets of pixel values representing a corresponding sequence of images, said sequence of sets of pixel values including a third set of pixel values and a fourth set of pixel values;

determines a sequence of displacements, wherein each displacement is of image relative to a preceding image in said sequence of images;

checks whether said sequence of displacements follow a uniform pattern; and

shift said window by said sequence of displacements in displaying corresponding portions of the images only if said sequence of displacements do not follow said uniform pattern.

15. A video camera designed to provide stable video images of a scene, said video camera comprising:

an image sensor designed to receive light representing an image of said scene, and generating a plurality of frames including a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances; and

an image processor operable to;

receive said first source frame and said second source frame;

generate a first subsampled frame from said first source frame, and a second subsampled frame from said second source frame;

compute a displacement of said second image relative to said first image by examining said first subsampled frame and said second subsampled frame; and

shift said second source frame by opposite of said displacement.

16. The video camera of claim 15, wherein said image processor is further operable to:

estimate a coarse displacement based on image content represented by said first subsampled image and said second subsampled image; and

determine said displacement with more accuracy than said coarse displacement by comparing said first source image and said second source image.

17. A video camera designed to provide stable video images of a scene, said video camera comprising:

an image sensor designed to receive light representing an image of said scene, and generating a plurality of frames including a first source frame and a second source frame, wherein said first source frame and said second source frame respectively represent a first image and a second image of a scene captured at different time instances; and

an image processor operable to;

receive a first set of pixel values representing a first image;

display a first area which covers only a first portion of said first image;

receive a second set of pixel values representing a second image;

determine a displacement of said second image relative to said first image; and

display a second area which covers only a second portion of said second image, wherein said second area is shifted by opposite of said displacement in relation to said first area.

18. The video camera of claim 17, wherein said image processor is operable to maintain a window which specifies the portion of images displayed by said displaying, wherein said window is moved by opposite of said displacement prior to displaying of said second area.

19. The video camera of claim 18, wherein said window, said first image, said second image, said first area and said second area are all rectangles.

20. The video camera of claim 19, wherein said image processor is further operable to:

receive a sequence of sets of pixel values following said second set of pixel values, said sequence of sets of pixel values representing a corresponding sequence of images, said sequence of sets of pixel values including a third set of pixel values and a fourth set of pixel values;

determine a sequence of displacements, wherein each displacement is of image relative to a preceding image in said sequence of images;

check whether said sequence of displacements follow a uniform pattern; and

shift said window by said sequence of displacements in displaying corresponding portions of the images only if said sequence of displacements do not follow said uniform pattern.