Positional sensor-assisted motion filtering for panoramic photography

• US 8,957,944 B2
• Filed: 05/17/2011
• Issued: 02/17/2015
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank, Zhou, Jianping
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): MIKESKA, NEIL R
• Application Number:
  US13/109,883
• Publication Number:
  US 20120293609A1
• Time in Patent Office: 1,372 Days
• Field of Search: 348/36, 382/284
• US Class Current: 348/36
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

This disclosure pertains to devices, methods, and computer readable media for perforating positional sensor-assisted panoramic photography techniques in handheld personal electronic devices. Generalized steps that may be used to carry out the panoramic photography techniques described herein include, but are not necessarily limited to: 1.) acquiring image data from the electronic device'"'"'s image sensor; 2.) performing “motion filtering” on the acquired image data, e.g., using information returned from positional sensors of the electronic device to inform the processing of the image data; 3.) performing image registration between adjacent captured images; 4.) performing geometric corrections on captured image data, e.g., due to perspective changes and/or camera rotation about a non-center of perspective (COP) camera point; and 5.) “stitching” the captured images together to create the panoramic scene, e.g., blending the image data in the overlap area between adjacent captured images. The resultant stitched panoramic image may be cropped before final storage.

Claims

1. An image processing method, comprising:

receiving a first stream of images from an image sensor in a device;

receiving rotational information for the device, wherein the rotational information comprises information related to the rotation of the device during the capture of the first stream of images;

storing, based at least in part on the rotational information, a subset of the first stream of images to generate a second sequence of images, wherein the act of storing occurs in response to the rotational information indicating that the device has rotated more than a threshold amount;

discarding the remainder of the first stream of images that are not a part of the subset;

combining a portion of each of the images comprising the second sequence of images so that each combined portion of each image in the second sequence of images overlaps at least one other portion of one other image in the second sequence of images; and

blending each of the overlaps between the portions of the images in the second sequence of images to produce a panoramic image,

wherein the rate at which the act of storing occurs changes dynamically during the receiving of the first stream of images in response to an amount of rotation imparted to the device.

2. The method of claim 1, further comprising the act of time syncing the received rotational information with the received first stream of images.

3. The method of claim 1, wherein the act of receiving rotational information comprises receiving rotational information from a gyrometer in the device.

4. The method of claim 1, further comprising receiving acceleration information for the device.

5. The method of claim 4, wherein the act of receiving acceleration information comprises receiving acceleration information from an accelerometer.

6. The method of claim 4, wherein the act of storing an image from the first stream of images for inclusion in the second sequence of images is based, at least in part, on the rotational information and the acceleration information.

7. The method of claim 1, wherein the act of combining generates alignment information for each combined pair of portions of each of the second sequence of images.

8. The method of claim 7, wherein the act of storing an image from the first stream of images for inclusion in the second sequence of images is based, at least in part, on the rotational information and the alignment information.

9. The method of claim 6, wherein the rate at which the act of storing occurs is directly related to the rate at which the device is being accelerated.

10. The method of claim 6, wherein the rate at which the act of storing occurs is directly related to the rate at which the device is being rotated.

11. The method of claim 7, wherein at least a portion of the alignment information is used to adjust the value of the threshold amount.

12. The method of claim 1, wherein the act of storing further comprises:

determining a rotation amount corresponding to the rotational information; and

searching a look up table for the rotation amount and identifying a corresponding number of images from the first stream of images that may be discarded before storing the next image from the first stream of images for inclusion in the second sequence of images.

13. The method of claim 1, wherein the act of storing further comprises:

determining a rotation amount corresponding to the rotational information; and

identifying, based at least in part on the rotation amount, a number of images from the first stream of images that may be discarded before storing the next image from the first stream of images for inclusion in the second sequence of images.

14. A program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 1.

15. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 1.

16. A method for generating a panoramic image from a plurality of images taken at a device, the method comprising:

receiving a first image, wherein the first image is taken with the device at a first position;

receiving data indicative of device movement from the first position to a second position;

filtering out a second image from the plurality of images based, at least in part, on a determination that the device'"'"'s movement from the first position to the second position has not exceeded a threshold amount of movement, wherein the second image is taken at the second position;

receiving data indicative of device movement from the second position to a third position;

receiving a third image from the plurality of images, wherein the third image is taken with the device at the third position;

performing image registration between the first image and the third image based, at least in part, on a determination that the device'"'"'s movement from the first position to the third position has exceeded the threshold amount of movement; and

generating the panoramic image using portions of at least the first image and the third image,

wherein the rate at which the act of filtering occurs changes dynamically during the receiving of the plurality of images in response to an amount of rotation imparted to the device.

17. The method of claim 16, further comprising:

determining a correspondence between the received data indicative of device movement and one or more images from the plurality of images.

18. The method of claim 16, wherein the device movement comprises rotation around an axis.

19. The method of claim 18, further comprising determining an amount of the rotation.

20. The method of claim 19, wherein determining the amount of rotation comprises determining an average amount of rotation over a first plurality of images taken after the first image in the plurality of images.

21. The method of claim 16, wherein the received data comprises rotational information received from a gyrometer in the device.

22. The method of claim 21, wherein the received data further comprises acceleration information received from an accelerometer in the device.

23. The method of claim 22, wherein the act of filtering out a second image from the plurality of images is based, at least in part, on the rotational information and the acceleration information.

24. The method of claim 16, wherein the act of performing image registration generates alignment information for the images that are registered.

25. The method of claim 16, wherein the rate at which the act of filtering occurs is directly related to the rate at which the device is being accelerated during the capture of the plurality of images.

26. The method of claim 16, wherein the rate at which the act of filtering occurs is directly related to the rate at which the device is being rotated during the capture of the plurality of images.

27. The method of claim 24, wherein at least a portion of the alignment information is used to adjust the value of the threshold amount.

28. The method of claim 16, wherein the act of filtering further comprises:

determining a rotation amount corresponding to the received data; and

searching a look up table for the rotation amount and identifying a corresponding number of images from the plurality of images that may be filtered before selecting the next image from the plurality of images for use in the generation of the panoramic image.

29. The method of claim 16, wherein the act of filtering further comprises:

determining a rotation amount corresponding to the received data; and

identifying, based at least in part on the rotation amount, a corresponding number of images from the plurality of images that may be filtered before selecting the next image from the plurality of images for use in the generation of the panoramic image.

30. A program storage device, readable by a programmable control device, comprising instructions stored thereon for causing the programmable control device to perform the method of claim 16.

31. An electronic device, comprising:

memory;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform the method of claim 16.