Positional sensor-assisted perspective correction for panoramic photography

• US 9,762,794 B2
• Filed: 05/17/2011
• Issued: 09/12/2017
• Est. Priority Date: 05/17/2011
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank, Zhou, Jianping
• Primary Examiner(s): MIKESKA, NEIL R
• Application Number:
  US13/109,878
• Publication Number:
  US 20120293608A1
• Time in Patent Office: 2,310 Days
• Field of Search: 348 36, 3482181, 3482221, 382248
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

This disclosure pertains to devices, methods, and computer readable media for performing positional sensor-assisted panoramic photography techniques in handheld personal electronic devices. Generalized steps that may be used to carry out the panoramic photography techniques described herein include, but are not necessarily limited to: 1.) acquiring image data from the electronic device'"'"'s image sensor; 2.) performing “motion filtering” on the acquired image data, e.g., using information returned from positional sensors of the electronic device to inform the processing of the image data; 3.) performing image registration between adjacent captured images; 4.) performing geometric corrections on captured image data, e.g., due to perspective changes and/or camera rotation about a non-center of perspective (COP) camera point; and 5.) “stitching” the captured images together to create the panoramic scene, e.g., blending the image data in the overlap area between adjacent captured images. The resultant stitched panoramic image may be cropped before final storage.

Claims

1. An image processing method, comprising:

obtaining a first image at a first time from a device comprising a lens and an image sensor;

obtaining a second image at a second time from the device;

receiving positional information from a sensor in the device, the positional information indicating an amount of change in a position of the device between the first and second times;

dividing the first image into a first plurality of image segments;

dividing the second image into a second plurality of image segments;

generating a first plurality of search vectors for the first plurality of image segments, wherein each of the first plurality of search vectors indicates an amount transformation to be applied to one of the first plurality of image segments to provide the image segment'"'"'s location in the second image;

generating a second plurality of search vectors for the second plurality of image segments, wherein each of the second plurality of search vectors indicates an amount transformation to be applied to one of the second plurality of image segments to provide the image segment'"'"'s location in the first image;

determining that one or more search vectors from the first and second pluralities of search vectors is consistent with the positional information, wherein the determination includes ignoring any search vector from the first and second pluralities of search vectors that is inconsistent with the positional information;

applying an amount of perspective correction to either the first or the second image; and

registering, in response to the applied perspective correction, the first image with the second image based on the determination.

2. The method of claim 1, wherein the positional information comprises rotational information indicating an amount of rotation of the device between the first and second times.

3. The method of claim 1, wherein obtaining a first image at a first time from a device comprises storing the first image in a memory.

4. The method of claim 1, wherein obtaining a second image at a second time from the device comprises storing the second image in a memory.

5. The method of claim 1, further comprising:

stitching the first image and second image together to create a third image,

wherein the third image comprises;

a first region corresponding to a first portion of only the first image,

an overlap region corresponding to a second portion of the first image overlapped with a first portion of the second image, and

a third region corresponding to a second portion of only the second image.

6. The method of claim 5, further comprising blending together the first and second images in the overlap region.

7. The method of claim 6, further comprising:

cropping the third image; and

storing the cropped third image in a memory.

8. The method of claim 1, wherein receiving positional information from a sensor in the device comprises receiving rotational information from a gyrometer.

9. The method of claim 8, wherein receiving positional information from a sensor in the device further comprises receiving device motion information from an accelerometer.

10. The method of claim 1, wherein applying an amount of perspective correction further comprises performing, on either the first or the second image, one or more of:

a cubic interpolation, a bicubic interpolation, a cubic spline, and a bicubic spline.

11. The method of claim 1, wherein applying an amount of perspective correction further comprises applying an image projection correction to either the first or the second image.

12. The method of claim 2, wherein the amount of rotation of the device between the first and second times is correlated to the amount of perspective correction to be applied to either the first or the second image.

13. The method of claim 12, wherein correlation comprises searching a look up table for the indicated rotation amount and identifying an amount of perspective correction corresponding to the rotation amount, and wherein the perspective correction is applied in accordance with the identified amount.

14. The method of claim 13, wherein the amount of perspective correction corresponding to a rotation amount in the look up table is determined based, at least in part, on a characteristic of the lens of the device.

17. The method of claim 1, wherein registering, in response to the applied perspective correction, the first image with the second image based on the determination comprises:

registering, in response to the applied perspective correction, the first image with the second image using the one or more search vectors that are consistent with the positional information and without using any of the ignored search vectors.

18. The method of claim 1, wherein registering, in response to the applied perspective correction, the first image with the second image based on the determination comprises:

determining that a number of ignored search vectors have a predetermined relationship with a threshold number of ignored search vectors; and

registering, in response to the applied perspective correction, the first image with the second image without using the one or more search vectors that are consistent with the positional information and without using any of the ignored search vectors.

19. The method of claim 1, wherein determining that one or more search vectors from the first and second pluralities of search vectors is consistent with the positional information comprises:

determining that one or more search vectors from the first and second pluralities of search vectors is consistent with a direction of movement indicated by the positional information.

20. The method of claim 1, wherein the change in the position of the device is caused by a user of the device.

21. The method of claim 1, wherein the amount of perspective correction is based, at least in part, on one or more of:

a focus level of the lens,

a resolution of the image sensor, and

the received positional information.

15. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon, the instructions comprising instructions for causing the programmable control device to:

obtain a first image at a first time from a device comprising a lens and an image sensor;

obtain a second image at a second time from the device;

receive positional information from a sensor in the device, the positional information indicating an amount of change in a position of the device between the first and second times;

divide the first image into a first plurality of image segments;

divide the second image into a second plurality of image segments;

generate a first plurality of search vectors for the first plurality of image segments, wherein each of the first plurality of search vectors indicates an amount transformation to be applied to one of the first plurality of image segments to provide the image segment'"'"'s location in the second image;

generate a second plurality of search vectors for the second plurality of image segments, wherein each of the second plurality of search vectors indicates an amount transformation to be applied to one of the second plurality of image segments to provide the image segment'"'"'s location in the first image;

determine that one or more search vectors from the first and second pluralities of search vectors is consistent with the positional information, wherein the determination includes ignoring any search vector from the first and second pluralities of search vectors that is inconsistent with the positional information;

apply an amount of perspective correction to either the first or the second image; and

register, in response to the applied perspective correction, the first image with the second image based on the determination.

22. The non-transitory program storage device of claim 15, wherein the positional information comprises rotational information indicating an amount of rotation of the device between the first and second times.

23. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to obtain a first image at a first time from a device comprise instructions for causing the programmable control device to store the first image in a memory.

24. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to obtain a second image at a second time from the device comprise instructions for causing the programmable control device to store the second image in a memory.

25. The non-transitory program storage device of claim 15, wherein the instructions comprise additional instructions for causing the programmable control device to:

stitch the first image and second image together to create a third image,

wherein the third image comprises;

a first region corresponding to a first portion of only the first image,

an overlap region corresponding to a second portion of the first image overlapped with a first portion of the second image, and

a third region corresponding to a second portion of only the second image.

26. The non-transitory program storage device of claim 25, wherein the instructions comprise additional instructions for causing the programmable control device to:

blend together the first and second images in the overlap region.

27. The non-transitory program storage device of claim 26, wherein the instructions comprise additional instructions for causing the programmable control device to:

crop the third image; and

store the cropped third image in a memory.

28. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to receive positional information from a sensor in the device comprise instructions for causing the programmable control device to receive rotational information from a gyrometer.

29. The non-transitory program storage device of claim 28, wherein the instructions for causing the programmable control device to receive positional information from a sensor in the device further comprise instructions for causing the programmable control device to receive device motion information from an accelerometer.

30. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to apply an amount of perspective correction further comprise instructions for causing the programmable control device to perform, on either the first or the second image, one or more of:

a cubic interpolation, a bicubic interpolation, a cubic spline, and a bicubic spline.

31. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to apply an amount of perspective correction further comprise instructions for causing the programmable control device to apply an image projection correction to either the first or the second image.

32. The non-transitory program storage device of claim 22, wherein the amount of rotation of the device between the first and second times is correlated to the amount of perspective correction to be applied to either the first or the second image.

33. The non-transitory program storage device of claim 32, wherein correlation comprises instructions for causing the programmable control device to search a look up table for the indicated rotation amount and identifying an amount of perspective correction corresponding to the rotation amount, and wherein the perspective correction is applied in accordance with the identified amount.

34. The non-transitory program storage device of claim 33, wherein the amount of perspective correction corresponding to a rotation amount in the look up table is determined based, at least in part, on a characteristic of the lens of the device.

35. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to register, in response to the applied perspective correction, the first image with the second image based on the determination comprise instructions for causing the programmable control device to:

register, in response to the applied perspective correction, the first image with the second image using the one or more search vectors that are consistent with the positional information and without using any of the ignored search vectors.

36. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to register, in response to the applied perspective correction, the first image with the second image based on the determination comprise instructions for causing the programmable control device to:

determine that a number of ignored search vectors have a predetermined relationship with a threshold number of ignored search vectors; and

register, in response to the applied perspective correction, the first image with the second image without using the one or more search vectors that are consistent with the positional information and without using any of the ignored search vectors.

37. The non-transitory program storage device of claim 15, wherein the instructions for causing the programmable control device to determine that one or more search vectors from the first and second pluralities of search vectors is consistent with the positional information comprise instructions for causing the programmable control device to:

determine that one or more search vectors from the first and second pluralities of search vectors is consistent with a direction of movement indicated by the positional information.

38. The non-transitory program storage device of claim 15, wherein the change in the position of the device is caused by a user of the device.

39. The non-transitory program storage device of claim 15, wherein the amount of perspective correction is based, at least in part, on one or more of:

a focus level of the lens,

a resolution of the image sensor, and

the received positional information.

16. An electronic device, comprising:

memory storing data, the data comprising instructions;

a lens;

an image sensor;

a positional sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, positional sensor, and image sensor, wherein the programmable control device is configured to execute one or more of the instructions to;

obtain a first image at a first time from the image sensor;

obtain a second image at a second time from the image sensor;

receive positional information from the positional sensor, the positional information indicating an amount of change in a position of the device between the first and second times;

divide the first image into a first plurality of image segments;

divide the second image into a second plurality of image segments;

generate a first plurality of search vectors for the first plurality of image segments, wherein each of the first plurality of search vectors indicates an amount transformation to be applied to one of the first plurality of image segments to provide the image segment'"'"'s location in the second image;

generate a second plurality of search vectors for the second plurality of image segments, wherein each of the second plurality of search vectors indicates an amount transformation to be applied to one of the second plurality of image segments to provide the image segment'"'"'s location in the first image;

determine that one or more search vectors from the first and second pluralities of search vectors is consistent with the positional information, wherein the determination includes ignoring any search vector from the first and second pluralities of search vectors that is inconsistent with the positional information;

apply an amount of perspective correction to either the first or the second image; and

register, in response to the applied perspective correction, the first image with the second image based on the determination.

40. The electronic device of claim 16, wherein the positional information comprises rotational information indicating an amount of rotation of the device between the first and second times.

41. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to obtain a first image at a first time from a device comprises the programmable control device being configured to execute one or more of the instructions to store the first image in a memory.

42. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to obtain a second image at a second time from the device comprises the programmable control device being configured to execute one or more of the instructions to store the second image in a memory.

43. The electronic device of claim 16, wherein the instructions comprise additional instructions and wherein the programmable control device is further configured to execute one or more of the additional instructions to:

stitch the first image and second image together to create a third image,

wherein the third image comprises;

a first region corresponding to a first portion of only the first image,

an overlap region corresponding to a second portion of the first image overlapped with a first portion of the second image, and

a third region corresponding to a second portion of only the second image.

44. The electronic device of claim 43, wherein the instructions comprise additional instructions and wherein the programmable control device is further configured to execute one or more of the additional instructions to:

blend together the first and second images in the overlap region.

45. The electronic device of claim 44, wherein the instructions comprise additional instructions and wherein the programmable control device is further configured to execute one or more of the additional instructions to:

crop the third image; and

store the cropped third image in a memory.

46. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to receive positional information from a sensor in the device comprises the programmable control device being configured to execute one or more of the instructions to receive rotational information from a gyrometer.

47. The electronic device of claim 46, wherein the programmable control device being configured to execute one or more of the instructions to receive positional information from a sensor in the device further comprises the programmable control device being configured to execute one or more of the instructions to receive device motion information from an accelerometer.

48. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to apply an amount of perspective correction further comprises the programmable control device being configured to execute one or more of the instructions to perform, on either the first or the second image, one or more of:

a cubic interpolation, a bicubic interpolation, a cubic spline, and a bicubic spline.

49. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to apply an amount of perspective correction further comprises the programmable control device being configured to execute one or more of the instructions to apply an image projection correction to either the first or the second image.

50. The electronic device of claim 40, wherein the amount of rotation of the device between the first and second times is correlated to the amount of perspective correction to be applied to either the first or the second image.

51. The electronic device of claim 50, wherein correlation comprises the programmable control device being configured to execute one or more of the instructions to search a look up table for the indicated rotation amount and identifying an amount of perspective correction corresponding to the rotation amount, and wherein the perspective correction is applied in accordance with the identified amount.

52. The electronic device of claim 51, wherein the amount of perspective correction corresponding to a rotation amount in the look up table is determined based, at least in part, on a characteristic of the lens of the device.

53. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to register, in response to the applied perspective correction, the first image with the second image based on the determination comprises the programmable control device being configured to execute one or more of the instructions to:

register, in response to the applied perspective correction, the first image with the second image using the one or more search vectors that are consistent with the positional information and without using any of the ignored search vectors.

54. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to register, in response to the applied perspective correction, the first image with the second image based on the determination comprises the programmable control device being configured to execute one or more of the instructions to:

determine that a number of ignored search vectors have a predetermined relationship with a threshold number of ignored search vectors; and

register, in response to the applied perspective correction, the first image with the second image without using the one or more search vectors that are consistent with the positional information and without using any of the ignored search vectors.

55. The electronic device of claim 16, wherein the programmable control device being configured to execute one or more of the instructions to determine that one or more search vectors from the first and second pluralities of search vectors is consistent with the positional information comprises the programmable control device being configured to execute one or more of the instructions to:

determine that one or more search vectors from the first and second pluralities of search vectors is consistent with a direction of movement indicated by the positional information.

56. The electronic device of claim 16, wherein the change in the position of the device is caused by a user of the device.

57. The electronic device of claim 16, wherein the amount of perspective correction is based, at least in part, on one or more of:

a focus level of the lens,

a resolution of the image sensor, and

the received positional information.