Interleaving image processing and image capture operations

• US 9,350,916 B2
• Filed: 05/28/2013
• Issued: 05/24/2016
• Est. Priority Date: 05/28/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Pflughaupt, Russell, Toft, Rolf, Hubel, Paul, Schulze, Anita Nariani
• Primary Examiner(s): Ho, Tuan
• Assistant Examiner(s): Wu, Zhenzhen
• Application Number:
  US13/903,671
• Publication Number:
  US 20140354842A1
• Time in Patent Office: 1,092 Days
• Field of Search: 348/239, 348/208.1
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions

Abstract

Systems, methods, and computer readable media to improve image stabilization operations and other image processing operations are described. A novel combination of interleaved image capture and image processing operations, e.g., image registration operations, may be employed on a bracketed capture of still images. Such techniques may result in improved camera performance and processing efficiency, as well as decreased shot-to-shot time intervals. In another embodiment, an image fusion portion of an image post-processing pipeline may also be performed in an interleaved fashion, such that each image in the sequence of obtained bracketed images may be incrementally added to an output composite image after it has been aligned with the preceding image or images from the sequence.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more programmable control devices to:

capture a first and a second image of a temporal sequence of images of a scene, each image having an associated capture time;

determine, prior to capture of a third image of the temporal sequence of images, a composite image contribution value for the first image and a first relationship between the second and first images;

capture the third image of the temporal sequence of images;

determine, prior to capture of a fourth image of the temporal sequence of images, a composite image contribution value for the second image and a second relationship between the third and second images;

register the first, second and third images based on the first and second relationships; and

fuse the first, second and third images based on their registration and each image'"'"'s composite image contribution value.

2. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to capture a first, second and third image of a temporal sequence of images of a scene comprise instructions to cause the one or more programmable control devices to obtain a bracketed image capture.

3. The non-transitory program storage device of claim 2, wherein the bracketed image capture is obtained by the one or more programmable control devices from an image capture device that is communicatively coupled to at least one of the one or more programmable control devices.

4. The non-transitory program storage device of claim 2, wherein the bracketed image capture comprises a High Dynamic Range (HDR) image.

5. The non-transitory program storage device of claim 2, wherein the bracketed image capture comprises a stabilized still image.

6. The non-transitory program storage device of claim 1, wherein at least one of the first, second and third images comprises a reference image.

7. The non-transitory program storage device of claim 6, further comprising instructions stored thereon to cause one or more programmable control devices to use the reference image for initialization purposes.

8. The non-transitory program storage device of claim 1, wherein the first and second images comprise consecutively obtained images from the temporal sequence of images.

9. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more programmable control devices to determine a first relationship between the second and first images comprise instructions to cause the one or more programmable control devices to match corresponding features between second and first images from the temporal sequence of images.

10. A method comprising:

capturing a first and a second image of a temporal sequence of images of a scene, each image having an associated capture time;

determining, prior to capture of a third image of the temporal sequence of images, a composite image contribution value for the first image and a first relationship between the second and first images;

capturing the third image of the temporal sequence of images;

determining, prior to capture of a fourth image of the temporal sequence of images, a composite image contribution value for the second image and a second relationship between the third and second images;

registering the first, second and third images based on the first and second relationships; and

fusing the first, second and third images based on their registration and each image'"'"'s composite image contribution value.

11. The method of claim 10, wherein capturing a first, second and third image of a temporal sequence of images of a scene comprises obtaining a bracketed image capture.

12. The method of claim 11, wherein the bracketed image capture comprises a plurality of images used to produce a stabilized still image.

13. The method of claim 10, wherein determining a first relationship between the second and first images comprises matching features between the second and first images from the temporal sequence of images.

14. A system comprising:

an image capture device;

memory; and

one or more programmable control devices operable to interact with the image capture device and the memory, and to perform operations comprising;

capturing a first and a second image of a temporal sequence of images of a scene with the image capture device, each image having an associated capture time;

determining, prior to capture of a third image of the temporal sequence of images, a composite image contribution value for the first image and a first relationship between the second and first images;

capturing the third image of the temporal sequence of images;

determining, prior to capture of a fourth image of the temporal sequence of images, a composite image contribution value for the second image and a second relationship between the third and second images;

registering the first, second and third images based on the first and second relationships;

fusing the first, second and third images based on their registration and each image'"'"'s composite image contribution value; and

storing the fused image of the scene in the memory.

15. The system of claim 14, wherein capturing a first, second and third image of a temporal sequence of images of a scene comprises obtaining a bracketed image capture.

16. The method of claim 15, further comprising generating a stabilized still image based on the bracketed image capture.

17. the system of claim 14, determining a first relationship between the second and first images comprises equalizing colors of the first image.

18. The system of claim 14, wherein determining a first relationship between the second and first images comprises matching features between the second and first images from the temporal sequence of images.

19. The system of claim 18, wherein determining a first relationship between the second and first images further comprises equalizing colors of the first image.