Intelligent image blending for panoramic photography

• US 9,088,714 B2
• Filed: 05/17/2011
• Issued: 07/21/2015
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank, Brunner, Ralph
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): ABIMBOLA, KEHINDE
• Application Number:
  US13/109,941
• Publication Number:
  US 20120293610A1
• Time in Patent Office: 1,526 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

This disclosure pertains to devices, methods, and computer readable media for performing positional sensor-assisted panoramic photography techniques in handheld personal electronic devices. Generalized steps that may be used to carry out the panoramic photography techniques described herein include, but are not necessarily limited to: 1.) acquiring image data from the electronic device'"'"'s image sensor; 2.) performing “motion filtering” on the acquired image data, e.g., using information returned from positional sensors of the electronic device to inform the processing of the image data; 3.) performing image registration between adjacent captured images; 4.) performing geometric corrections on captured image data, e.g., due to perspective changes and/or camera rotation about a non-center of perspective (COP) camera point; and 5.) “stitching” the captured images together to create the panoramic scene, e.g., blending the image data in the overlap area between adjacent captured images. The resultant stitched panoramic image may be cropped before final storage.

Claims

1. A method to generate a panoramic image, comprising:

obtaining a first image from an image sensor having a first region and a second region, the second region including a first representation of a face, the first image stored in a first memory;

obtaining a second image from the image sensor having a third region and a fourth region, the third region including a second representation of the face, the second image stored in a second memory;

aligning the first and second images so that the second and third regions overlap to generate an overlap region;

masking an area corresponding to the first representation of the face in the overlap region to generate a mask region;

blending the first and second images in the overlap region, except for the mask region, to generate a blended region; and

generating a result image comprising the first region, the fourth region and the blended region wherein the area in the blended region corresponding to the mask region is replaced with the first representation of the face.

2. The method of claim 1, further comprising storing the result image in a third memory.

3. The method of claim 1, wherein the first representation of a face comprises a representation of a complete face.

4. The method of claim 1, wherein the act of aligning the first and second images comprises using feature detection.

5. The method of claim 1, wherein the act of aligning the first and second images comprises using cross-correlation.

6. The method of claim 1, wherein the act of blending the first and second images in the blended region comprises using an alpha blending scheme.

7. The method of claim 1, wherein the act of obtaining a first image comprises:

obtaining a first image having a first region and a second region, wherein the second region includes a plurality of representations, each representation corresponding to a face; and

selecting one of the plurality of representations as the first representation of the face.

8. The method of claim 7, wherein the act of selecting one of the plurality of representations comprises selecting the largest representation of a face to be the first representation of the face.

9. The method of claim 1, wherein the act of obtaining the first image comprises capturing the first image.

10. The method of claim 1, wherein the act of aligning the first and second images further comprises not overlapping the first and fourth regions.

11. The method of claim 1, wherein the act of masking an area corresponding to the first representation of the face to generate a mask region comprises:

using image metadata to identify a first area corresponding to the first representation of the face; and

expanding the first area in at least one direction to generate a mask region.

12. The method of claim 1, wherein the act of masking an area corresponding to the first representation of the face to generate a mask region comprises:

generating a skin tone mask corresponding to the first representation of the face.

13. The method of claim 1, wherein the first and second images are high dynamic range (HDR) images.

14. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 1.

15. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 1.

16. A method to generate a panoramic image comprising:

(a) receiving data representative of a plurality of images comprising a scene from an image sensor;

(b) determining an overlapping region between a first image and a second image from the plurality of images;

(c) identifying a feature of interest that is represented at a location in each of the first image and the second image, wherein each of the representations are located in the overlapping region;

(d) selecting the representation of the feature of interest from the first image;

(e) blending between the first image and the second image in the overlapping region to generate a resulting overlapping region; and

(f) assembling the first image and the second image, using the resulting overlapping region to replace the overlapping region between the first image and the second image,

wherein the act of blending excludes the location of the identified feature of interest, and

wherein the selected representation is used in the resulting overlapping region at the location of the identified feature of interest.

17. The method of claim 16, wherein the act of determining the overlapping region comprises registering the first and second images.

18. The method of claim 17, wherein the act of registering comprises performing feature detection.

19. The method of claim 17, wherein the act of registering comprises performing cross-correlation.

20. The method of claim 17, wherein the act of registering comprises using positional information received from a positional sensor in communication with the image sensor.

21. The method of claim 16, wherein the feature of interest comprises a face.

22. The method of claim 21, wherein the act of selecting the representation of the face from either the first image or the second image is based at least in part on a calculated score of the representation of the face from the first image and a calculated score of the representation of the face from the second image.

23. The method of claim 22, wherein the calculation of the score of the representation of a face comprises a consideration of at least one of the following:

smile confidence for the face, a detection of red-eye artifacts within the face, and a detection of open or closed eyes within the face.

24. The method of claim 21, wherein the location comprises a skin tone mask created over the face.

25. The method of claim 24, wherein the skin tone mask has a feathered edge.

26. The method of claim 16, wherein the location comprises a bounding box circumscribing the feature of interest.

27. The method of claim 26, wherein the bounding box is padded by a determined number of pixels in at least one direction around the feature of interest.

28. The method of claim 16, wherein the act of blending comprises using an alpha blending scheme.

29. The method of claim 16, wherein the act of blending comprises using a linear blending function.

30. The method of claim 16, wherein the act of blending comprises using a polynomial blending function.

31. The method of claim 16, wherein steps (b)-(f) are repeated for every pair of images from the plurality of images having an overlapping region.

32. The method of claim 16, wherein the plurality of images are HDR images.

33. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 16.

34. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 16.

35. A method for generating a panoramic image, comprising:

receiving data representative of a plurality of images from an image sensor;

identifying one or more locations in the plurality of images in which one or more faces are located; and

blending overlapping regions of the plurality of images to form a panoramic image,

wherein the act of blending excludes regions of an image having the one or more face locations in an overlapping region.

36. The method of claim 35, wherein each excluded region in the panoramic image comprises an unblended representation of the face located in the excluded region, and wherein the unblended representation of the face is taken from one of the images represented in the overlapping region having the excluded region.

37. The method of claim 35, wherein the excluded regions comprise skin tone masks created over the located face.

38. The method of claim 37, wherein the skin tone mask has a feathered edge.

39. The method of claim 35, wherein the excluded regions comprise bounding boxes circumscribing the located faces.

40. The method of claim 39, wherein each bounding box is padded by a determined number of pixels in at least one direction around the located face.

41. The method of claim 35, wherein the act of blending comprises using an alpha blending scheme.

42. The method of claim 35, wherein the plurality of images are HDR images.

43. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 35.

44. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 35.