Pixel analysis and frame alignment for background frames

• US 8,254,760 B2
• Filed: 08/28/2009
• Issued: 08/28/2012
• Est. Priority Date: 08/28/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Neuman, Michael, Groszko, G. Garrett
• Primary Examiner(s): Chevalier, Robert
• Application Number:
  US12/550,298
• Publication Number:
  US 20110051991A1
• Time in Patent Office: 1,096 Days
• Field of Search: 386/278, 386/280, 386/286, 386/210, 386/224, 386/230, 386/242
• US Class Current: 386/278
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence

Abstract

Background frames can be completed from a collection of frames having foreground objects that are partially obscuring the pixels comprising the background. The special offset of a pixel represented across a collection of frames can be determined based on camera movement data. By determining the relative offset of a pixel represented in a first frame from the same pixel in other frames, pixel values representing a background object can be accumulated to derive a completed background frame.

Claims

1. A method for completing a background frame from two or more frames in a collection comprising:

selecting a first frame and one or more second frames from a collection of frames for analysis;

determining a spatial offset between the first and second frames;

aligning pixels of the second frames with the pixels of the first frame based on the spatial offset data;

analyzing the pixels comprising the first and second frames; and

maintaining predominant pixels values across the analyzed frames.

2. The method of claim 1, wherein the special offset is determined by analyzing frames to identify at least two clusters of pixels having little inter-frame change and determining the relative movement of at least one cluster of pixels against at least one other cluster of pixels.

3. The method of claim 1, further comprising:

calculating vectors for a plurality of pixels represented in the first frame and the one or more second frames, the vector representing the respective distance a pixel has moved from one frame to the next due to a change in camera position; and

identifying objects comprised of pixels having greater vectors as foreground objects and objects comprised of pixels having lesser vectors as background objects.

4. The method of claim 3, further comprising:

averaging the pixel values of those pixels identified as background objects and comparing pixel values from other frames against the average;

identifying pixels from other frames having approximately similar pixel values as also comprising background pixels; and

maintaining these pixel values across the analyzed frames.

5. The method of claim 1, wherein the frames comprise individual photographs.

6. A non-transitory computer-readable medium storing computer executable instructions for implementing a method on a device, the instructions comprising:

selecting a first frame and one or more second frames from a collection of frames for analysis;

determining a spatial offset between the first and second frames;

aligning pixels of the second frames with the pixels of the first frame based on the spatial offset data;

analyzing the pixels comprising the first and second frames; and

maintaining predominant pixels values across the analyzed frames.

7. The non-transitory computer-readable medium of claim 6, wherein the special offset is determined by analyzing frames to identify at least two clusters of pixels having little inter-frame change and determining the relative movement of at least one cluster of pixels against at least one other cluster of pixels.

8. The non-transitory computer-readable medium of claim 6, the instructions further comprising:

calculating vectors for a plurality of pixels represented in the first frame and the one or more second frames, the vector representing the respective distance a pixel has moved from one frame to the next due to a change in camera position; and

identifying objects comprised of pixels having greater vectors as foreground objects and objects comprised of pixels having lesser vectors as background objects.

9. The non-transitory computer-readable medium of claim 8, the instructions further comprising:

averaging the pixel values of those pixels identified as background objects and comparing pixel values from other frames against the average;

identifying pixels from other frames having approximately similar pixel values as also comprising background pixels; and

maintaining these pixel values across the analyzed frames.

10. A device comprising:

a video processor configured to select a first frame and one or more second frames from a collection of frames for analysis and determining a spatial offset between the first and second frames, the processor further configured for analyzing pixels comprising the first and second frames and comparing values corresponding to a pixel in a first frame with a pixel in the second frame, the pixel in the second frame being identified based on the special offset with respect to the location of the first pixel in the first frame, and configured to maintain predominant pixels values across the analyzed frames.