Positional Sensor-Assisted Image Registration for Panoramic Photography

• US 20120294549A1
• Filed: 05/17/2011
• Published: 11/22/2012
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank
• Granted Patent: US 8,600,194 B2
• Time in Patent Office: Days
• US Class Current: 382/294
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

This disclosure pertains to devices, methods, and computer readable media for performing positional sensor-assisted panoramic photography techniques in handheld personal electronic devices. Generalized steps that may be used to carry out the panoramic photography techniques described herein include, but are not necessarily limited to: 1.) acquiring image data from the electronic device'"'"'s image sensor; 2.) performing “motion filtering” on the acquired image data, e.g., using information returned from positional sensors of the electronic device to inform the processing of the image data; 3.) performing image registration between adjacent captured images; 4.) performing geometric corrections on captured image data, e.g., due to perspective changes and/or camera rotation about a non-center of perspective (COP) camera point; and 5.) “stitching” the captured images together to create the panoramic scene, e.g., blending the image data in the overlap area between adjacent captured images. The resultant stitched panoramic image may be cropped before final storage.

Claims

1. An image registration method, comprising:

obtaining positional information from a device;

obtaining first and second images from the device;

aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image to identify a plurality of corresponding regions;

determining a search vector for each of the plurality of corresponding regions;

selecting only those corresponding regions from the plurality of corresponding regions having a search vector consistent with the positional information to identify a plurality of consistent regions; and

registering the first and second images using the plurality of consistent regions.

2. The method of claim 1, wherein the act of obtaining positional information comprises obtaining positional information from a gyrometer in the device.

3. The method of claim 1, wherein the act of obtaining positional information comprises obtaining positional information from an accelerometer in the device.

4. The method of claim 1, wherein the act of obtaining first and second images comprises:

obtaining a first full size image;

selecting less that all of the first full size image to generate the first image;

obtaining a second full size image; and

selecting less that all of the second full size image to generate the second image.

5. The method of claim 1, wherein the act of aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image comprises identifying individual features in the first and second images using feature detection.

6. The method of claim 1, wherein the act of aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image comprises using cross-correlation.

7. The method of claim 1, wherein the act of aligning a plurality of regions in the first image with a corresponding plurality of regions in the second image to identify a plurality of corresponding regions comprises:

using feature detection if the act of selecting identifies a threshold number or more than a threshold number of consistent regions; and

using cross-correlation if the act of selecting identifies less than a threshold number of consistent regions.

8. The method of claim 1, further comprising using the obtained positional information if the act of using cross-correlation identifies less than a second threshold number of consistent regions.

9. The method of claim 1, wherein the act of determining a search vector for each of the plurality of corresponding regions comprises determining the direction a feature in the first image would need to move to align with a corresponding feature in the second image.

10. The method of claim 1, wherein the act of selecting only those corresponding regions from the plurality of corresponding regions having a search vector consistent with the positional information to identify a plurality of consistent regions comprises discarding regions having a search vector opposed to or substantially opposed to a direction of movement indicated by the positional information.

11. The method of claim 1, wherein the act of registering generates a panoramic image comprising the first and second images.

12. A program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 1.

13. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 1.

14. An image registration method, comprising:

receiving a first image captured by a device;

receiving device movement data from one or more positional sensors;

receiving a second image captured by a device; and

performing image registration on the second image using the device movement data and the first image,

wherein the device movement data provides a search vector used in the act of performing image registration, and

wherein the second image is captured by the device at a later point in time than the first image.

15. The method of claim 14, wherein the act of performing image registration comprises performing feature detection on the first and second image and matching corresponding detected features between the first and second images.

16. The method of claim 14, wherein the act of performing image registration comprises cross-correlating intensity patterns in the first image with intensity patterns in the second image via correlation metrics.

17. The method of claim 14, wherein the act of performing image registration comprises:

performing feature detection, and, if sufficient data is not generated via the performance of feature detection to perform image registration, then

performing cross-correlation, and, if sufficient data is not generated via the performance of cross-correlation to perform image registration, then

using the received device movement data to perform image registration.

19. The method of claim 14, wherein the act of receiving device movement data from one or more positional sensors comprises obtaining information from an accelerometer in the device.

23. A program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 14.

24. The program storage device of claim 23, further comprising instructions stored thereon for causing the programmable control device to perform the method of claim 17.

25. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 14.

18. The method of dam 14, wherein the act of receiving device movement data from one or more positional sensors comprises obtaining information from a gyrometer in the device.

20. The method of dam 14, wherein the device movement data further provides a search vector for each of a plurality of regions comprising the second image.

21. The method of dam 20, wherein each search vector is indicative of a direction that a feature or intensity pattern in the first image would need to move to align with the corresponding feature or intensity pattern in the second image.

22. The method of dam 21, wherein the act of performing image registration comprises discarding those regions from the plurality of regions comprising the second image having a search vector opposed to or substantially opposed to a direction of movement indicated by the received device movement data.