Positional sensor-assisted image registration for panoramic photography

• US 8,600,194 B2
• Filed: 05/17/2011
• Issued: 12/03/2013
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US13/109,889
• Publication Number:
  US 20120294549A1
• Time in Patent Office: 931 Days
• Field of Search: 382/284, 382/294, 382/296, 382/305, 382/312, 345/629, 345/651, 348/36, 358/540
• US Class Current: 382/294
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

This disclosure pertains to devices, methods, and computer readable media for performing positional sensor-assisted panoramic photography techniques in handheld personal electronic devices. Generalized steps that may be used to carry out the panoramic photography techniques described herein include, but are not necessarily limited to: 1.) acquiring image data from the electronic device'"'"'s image sensor; 2.) performing “motion filtering” on the acquired image data, e.g., using information returned from positional sensors of the electronic device to inform the processing of the image data; 3.) performing image registration between adjacent captured images; 4.) performing geometric corrections on captured image data, e.g., due to perspective changes and/or camera rotation about a non-center of perspective (COP) camera point; and 5.) “stitching” the captured images together to create the panoramic scene, e.g., blending the image data in the overlap area between adjacent captured images. The resultant stitched panoramic image may be cropped before final storage.

Claims

1. An image registration method, comprising:

obtaining positional information from one or more positional sensors of a device;

obtaining first and second images from the device;

aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image to identify a plurality of corresponding regions;

determining a search vector for each of the plurality of corresponding regions, wherein the act of determining a search vector comprises determining a direction that a feature in the first image would need to move to align with a corresponding feature in the second image;

selecting only those corresponding regions from the plurality of corresponding regions having a search vector consistent with the positional information to identify a plurality of consistent regions, wherein the act of identifying a plurality of consistent regions comprises discarding regions having a search vector opposed to or substantially opposed to a direction of movement indicated by the positional information; and

registering the first and second images using the plurality of consistent regions.

2. The method of claim 1, wherein the act of obtaining positional information comprises obtaining positional information from a gyrometer in the device.

3. The method of claim 1, wherein the act of obtaining positional information comprises obtaining positional information from an accelerometer in the device.

4. The method of claim 1, wherein the act of obtaining first and second images comprises:

obtaining a first full size image;

selecting less than all of the first full size image to generate the first image;

obtaining a second full size image; and

selecting less than all of the second full size image to generate the second image.

5. The method of claim 1, wherein the act of aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image comprises identifying individual features in the first and second images using feature detection.

6. The method of claim 1, wherein the act of aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image comprises using cross-correlation.

7. The method of claim 1, wherein the act of aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image to identify a plurality of corresponding regions comprises:

using feature detection if the act of selecting identifies a threshold number or more than a threshold number of consistent regions; and

using cross-correlation if the act of selecting identifies less than a threshold number of consistent regions.

8. The method of claim 1, further comprising using the obtained positional information if the act of using cross-correlation identifies less than a second threshold number of consistent regions.

9. The method of claim 1, wherein the act of registering generates a panoramic image comprising the first and second images.

10. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 1.

11. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 1.

12. An image registration method, comprising:

receiving a first image captured by a device;

receiving device movement data from one or more positional sensors of the device;

receiving a second image captured by a device; and

performing image registration on the second image using the device movement data and the first image,

wherein the device movement data provides a search vector used in the act of performing image registration,

wherein the search vector is indicative of a direction that a feature or intensity pattern in the first image would need to move to align with the corresponding feature or intensity pattern in the second image,

wherein the act of performing image registration comprises discarding those regions from the plurality of regions comprising the second image having a search vector opposed to or substantially opposed to a direction of movement indicated by the received device movement data, and

wherein the second image is captured by the device at a later point in time than the first image.

13. The method of claim 12, wherein the act of performing image registration comprises performing feature detection on the first and second image and matching corresponding detected features between the first and second images.

14. The method of claim 12, wherein the act of performing image registration comprises cross-correlating intensity patterns in the first image with intensity patterns in the second image via correlation metrics.

15. The method of claim 12, wherein the act of performing image registration comprises:

performing feature detection, and, if sufficient data is not generated via the performance of feature detection to perform image registration, then

performing cross-correlation, and, if sufficient data is not generated via the performance of cross-correlation to perform image registration, then

using the received device movement data to perform image registration.

16. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 12.

17. The method of claim 12, wherein the act of receiving device movement data from one or more positional sensors comprises obtaining information from a gyrometer in the device.

18. The method of claim 12, wherein the act of receiving device movement data from one or more positional sensors comprises obtaining information from an accelerometer in the device.

19. The method of claim 12, wherein the device movement data further provides a search vector for each of a plurality of regions comprising the second image.

20. The non-transitory program storage device of claim 16, further comprising instructions stored thereon for causing the programmable control device to perform the method of claim 15.

21. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 12.

22. The electronic device of claim 21, wherein the instructions to perform image registration further comprise instructions to perform feature detection on the first and second image and match corresponding detected features between the first and second images.

23. The electronic device of claim 21, wherein the instructions to perform image registration further comprise instructions to cross-correlate intensity patterns in the first image with intensity patterns in the second image via correlation metrics.

24. The electronic device of claim 21, wherein the instructions to perform image registration further comprise instructions to:

perform feature detection, and, if sufficient data is not generated via the performance of feature detection to perform image registration, then

perform cross-correlation, and, if sufficient data is not generated via the performance of cross-correlation to perform image registration, then

use the received device movement data to perform image registration.

25. The electronic device of claim 21, wherein the instructions to receive device movement data from one or more positional sensors further comprise instructions to obtain information from a gyrometer in the device.