Exposure mapping and dynamic thresholding for blending of multiple images using floating exposure

• US 9,832,378 B2
• Filed: 06/06/2013
• Issued: 11/28/2017
• Est. Priority Date: 06/06/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank, Tico, Marius, Chen, Ting, Liang, Yongfang
• Primary Examiner(s): Philippe, Gims
• Assistant Examiner(s): NASRI, MARYAM A
• Application Number:
  US13/911,243
• Publication Number:
  US 20140362173A1
• Time in Patent Office: 1,636 Days
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Special blend operations for wide area-of-view image generation utilizing a “floating auto exposure” scheme are described. Pixel values in the two images being stitched together are blended within a transition band around a “seam.” identified in the overlap region between the images after changes in exposure and/or color saturation are accounted for. In some embodiments, changes in exposure and/or color saturation are accounted for through the use of one or more exposure mapping curves, the selection and use of which are based, at least in part, on a determined “Exposure Ratio” value, i.e., the amount that the camera'"'"'s exposure settings have deviated from their initial capture settings. In other embodiments, the Exposure Ratio value is also used to determine regions along the seam where either: alpha blending, Poisson blending—or a combination of the two—should be used to blend in the transitional areas on each side of the seam.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:

obtain a first image of a scene from a camera at a first time using a first exposure value;

obtain a second image of the scene from the camera at a second, later time using a second exposure value, wherein the first and second images share an overlapping region, the overlapping region comprising a plurality of pixels from the first image and a corresponding plurality of pixels from the second image, each pixel having a pixel value comprising one or more of a luminance value and a chrominance value;

determine an exposure ratio value based on the first exposure value and the second exposure value;

determine an updated pixel value for one or more of the plurality of pixels in the second image based, at least in part, on the determined exposure ratio value;

update the pixel values of the one or more pixels from the plurality of pixels in the second image to be equal to the corresponding determined updated pixel values; and

combine at least some of the pixels having the updated pixel values from the second image with corresponding pixels from the first image to generate a wide area-of-view image.

2. The non-transitory program storage device of claim 1, wherein the exposure ratio value comprises a ratio of the second exposure value to the first exposure value.

3. The non-transitory program storage device of claim 1, wherein the instructions to cause the programmable control device to determine an updated pixel value for one or more of the plurality of pixels in the second image comprise instructions to cause the programmable control device to:

select an exposure mapping curve based, at least in part, on the determined exposure ratio value; and

apply the selected exposure mapping curve to at least some of the one or more pixels from the plurality of pixels in the second image.

4. The non-transitory program storage device of claim 3, wherein a first exposure mapping curve is selected when the second exposure value is larger than the first exposure value, and a second exposure mapping curve is selected when the second exposure value is smaller than the first exposure value.

5. The non-transitory program storage device of claim 1, wherein the one or more of the plurality of pixels in the second image for which an updated value is determined comprise one of more pixels located in a central slice of the second image.

6. The non-transitory program storage device of claim 1, further comprising instructions to cause the programmable control device to determine an upper limit exposure value and a lower limit exposure value of the camera in response to capturing the first image and before capturing the second image, wherein the upper limit exposure value and the lower limit exposure value are based on the first exposure value.

7. The non-transitory program storage device of claim 6, wherein the instructions to cause the programmable control device to determine an updated pixel value for one or more of the plurality of pixels in the second image further comprise instructions to cause the programmable control device to:

determine a desaturation scale metric based, at least in part, on the determined exposure ratio value and a calculated gain value; and

apply the desaturation scale metric to at least some of the one or more pixels in the second image.

8. The non-transitory program storage device of claim 7, wherein the calculated gain value comprises a difference between a first gain that was applied during a capture of the second image and a second gain that would have been applied during the capture of the second image when the second exposure value is at least equal to the lower limit exposure value and at most equal to the upper limit exposure value.

9. The non-transitory program storage device of claim 6, wherein the second exposure value is at least equal to the lower limit and at most equal to the upper limit.

10. A method, comprising:

obtaining a first image of a scene from a camera at a first time using a first exposure value;

obtaining a second image of the scene from the camera at a second, later time using a second exposure value, wherein the first and second images share an overlapping region, the overlapping region comprising a plurality of pixels from the first image and a corresponding plurality of pixels from the second image, each pixel having a pixel value comprising one or more of a luminance value and a chrominance value;

determining an exposure ratio value based on the first exposure value and the second exposure value;

determining an updated pixel value for one or more of the plurality of pixels in the second image based, at least in part, on the determined exposure ratio value;

updating the pixel values of the one or more pixels from the plurality of pixels in the second image to be equal to the corresponding determined updated pixel values; and

combining at least some of the pixels having the updated pixel values of the second image with corresponding pixels from the first image to generate a wide area-of-view image.

11. The method of claim 10, wherein the exposure ratio value comprises a ratio of the second exposure value to the first exposure value.

12. The method of claim 10, wherein determining an updated pixel value further comprises:

selecting an exposure mapping curve based, at least in part, on the determined exposure ratio value; and

applying the selected exposure mapping curve to at least some of the one or more pixels from the plurality of pixels in the second image.

13. The method of claim 12, wherein a first exposure mapping curve is selected when the second exposure value is larger than the first exposure value, and a second exposure mapping curve is selected when the second exposure value is smaller than the first exposure value.

14. The method of claim 10, further comprising:

determining an upper limit exposure value and a lower limit exposure value of the camera in response to capturing the first image and before capturing the second image, wherein the upper limit exposure value and the lower limit exposure value are based on the first exposure value.

15. The method of claim 14, wherein determining an updated pixel value for one or more of the plurality of pixels in the second image further comprises:

determining a desaturation scale metric based, at least in part, on the determined exposure ratio value and a calculated gain value; and

applying the desaturation scale metric to at least some of the one or more pixels in the second image.

16. The method of claim 15, wherein the calculated gain value comprises a difference between a first gain that was applied during a capture of the second image and a second gain that would have been applied during the capture of the second image when the second exposure value is at least equal to the lower limit exposure value and at most equal to the upper limit exposure value.

17. The method of claim 14, wherein the second exposure value is at least equal to the lower limit and at most equal to the upper limit.

18. A system, comprising:

an image capture device;

a memory operatively coupled to the image capture device and having, stored therein, computer program code;

a display element operatively coupled to the memory; and

a programmable control device operatively coupled to the memory and the display element and comprising instructions stored thereon to cause the programmable control device to;

obtain a first image of a scene from the image capture device at a first time using a first exposure value;

obtain a second image of the scene from the image capture device at a second, later time using a second exposure value, wherein the first and second images share an overlapping region, the overlapping region comprising a plurality of pixels from the first image and a corresponding plurality of pixels from the second, each pixel having a pixel value comprising one or more of a luminance value and a chrominance value;

determine an exposure ratio value based on the first exposure value and the second exposure value;

determine an updated pixel value for one or more of the plurality of pixels in the second image based, at least in part, on the determined exposure ratio value;

update the pixel values of the one or more pixels from the plurality of pixels in the second image to be equal to the corresponding determined updated pixel values;

combine at least some of the pixels having the updated pixel values from the second image with corresponding pixels from the first image to generate a wide area-of-view image; and

store the wide area-of-view image in the memory.

19. The system of claim 18, wherein the exposure ratio value comprises a ratio of the second exposure value to the first exposure value.

20. The system of claim 18, wherein the instructions to cause the programmable control device to determine an updated pixel value for one or more of the plurality of pixels in the second image comprise instructions to cause the programmable control device to:

select an exposure mapping curve based, at least in part, on the determined exposure ratio value; and

apply the selected exposure mapping curve to at least some of the one or more pixels from the plurality of pixels in the second image.

21. The system of claim 18, further comprising instructions to cause the programmable control device to determine an upper limit exposure value and a lower limit exposure value of the image capture device in response to capturing the first image and before capturing the second image, wherein the upper limit exposure value and the lower limit exposure value are based on the first exposure value.

22. The system of claim 21, wherein the second exposure value is at least equal to the lower limit and at most equal to the upper limit.

23. The system of claim 18, wherein the instructions to cause the programmable control device to determine an updated pixel value for one or more of the plurality of pixels in the second image further comprise instructions to cause the programmable control device to:

determine a desaturation scale metric based, at least in part, on the determined exposure ratio value and a calculated gain value; and

apply the desaturation scale metric to at least some of the one or more pixels in the second image.