Methods and apparatus for generating composite images

• US 9,343,043 B2
• Filed: 08/01/2013
• Issued: 05/17/2016
• Est. Priority Date: 08/01/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lowe, David, Clark, Geoffrey, Brown, Matthew, Beis, Jeffrey
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US13/957,298
• Publication Number:
  US 20150035857A1
• Time in Patent Office: 1,020 Days
• Field of Search: 382/103, 382/118, 382/255, 382/274, 382/254, 382/284, 348/222.1, 348/79, 348/239, 348/77, 348/92, 348/253, 348/169, 348/584, 345/629, 345/419, 345/630
• US Class Current: 1/1
• CPC Class Codes: G03B 13/02 Viewfinders
• CPC Class Codes: G03B 2217/18 Signals indicating condition of a camera member or suitability of light
• CPC Class Codes: G03B 37/02 with scanning movement of lens or cameras
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Apparatuses and methods for capturing and generating composite images. In various aspects, the invention provides apparatuses and methods for correcting position information of captured images received by position sensors based on alignment of overlapping images. Corrected position information is then taken into account when displaying the locations of captured images on a display for providing guidance to users for generating composite images.

Claims

1. An apparatus for capturing images used for generating a composite image of a scene, the apparatus comprising:

an image sensor for capturing images;

a position sensor for determining the apparatus'"'"'s position when the images are captured;

one or more processors configured to;

associate the apparatus'"'"' position when each of the images is captured with such image;

generate a composite image including a live image received from the image sensor and representations of the captured images superimposed onto the live image using the associated positions, wherein each representation of a captured image includes a frame defining an area of the captured image relative to a position on the live image, the representations indicating areas of a scene being depicted in the live image that have already been acquired as captured images;

provide the composite image to the display;

a display for displaying the composite image, the display having controls for manipulating the representations of the composite image.

2. An apparatus according to claim 1 wherein the position sensor comprises a gyroscope.

3. An apparatus according to claim 1 wherein the apparatus is a mobile computing device.

4. An apparatus according to claim 1 wherein the representations are displayed by the display with perspective distortion based on positioning of the apparatus.

5. An apparatus according to claim 1 wherein the one or more processors are further configured to align the captured images using SIFT.

6. An apparatus according to claim 5 wherein the one or more processors are configured to use the alignment information to correct the associated positions.

7. An apparatus according to claim 1 wherein the controls comprise interfaces for replacing or deleting one or all of the representations.

8. An apparatus according to claim 1 wherein the representations include image data of the captured images and the controls comprise opacity modifiers for adjusting opacity of the representations relative to the live image.

9. An apparatus according to claim 1 wherein the controls are superimposed on top of the composite image.

10. An apparatus according to claim 1 wherein the composite image is a panoramic image of the scene.

11. An apparatus according to claim 1 wherein the representations further comprise one or more markers indicating position of the captured image.

12. The apparatus of claim 1, wherein the representations are overlaid with a predetermined image treatment that makes the areas corresponding to the captured images visually distinctive from the live image.

13. A method for correcting position information of images captured with an imaging apparatus and displaying representations of the captured images, the method comprising:

matching, by one or more processors, features on the captured images having overlapping segments;

creating, by the one or more processors, a correction factor based on the matched features;

correcting, by the one or more processors, position information associated with each captured image based on the correction factor;

generating, by the one or more processors, a composite image including a live image received from an image sensor and representations of the captured images superimposed onto the live image using the corrected position information, wherein each representation of a captured image includes a frame defining an area of the captured image relative to a position on the live image, the representations indicating areas of a scene being depicted in the live image that have already been acquired as captured images;

providing, by the one or more processors, the composite image for display.

14. A method according to claim 13 wherein the position information comprises a 3-D rotation matrix that provides information about the roll, pitch, and yaw of the imaging apparatus when the image is captured.

15. A method according to claim 13 further comprising the steps of generating a bundle adjustment from a plurality of captured images and correcting the position information is further based on the bundle adjustment.

16. A method according to claim 13 further comprising the step of displaying the composite image on a display of the imaging apparatus with controls for manipulating the captured images.

17. A method according to claim 13 wherein SIFT is used for matching features in the captured images.

18. A method according to claim 13 wherein the correction factor is created by homography transformation.

19. A method according to claim 13 wherein the correction of position information occurs continuously as images are captured.

20. A method according to claim 13 wherein the correction of position information occurs iteratively.

21. A method according to claim 13 wherein the correction factor is a 3-D rotation matrix.

22. A method for capturing images using an imaging apparatus having a display for displaying a composite image of a scene, the method performed comprising:

associating, by one or more processors, position information of the imaging apparatus when at least two of the images are captured with the applicable image;

correcting, by one or more processors, position information of the at least two images based on overlap between the captured images;

using the position information to generate a representation for each of the at least two images, wherein each representation includes a frame defining an area one of the at least two images;

generating a composite image including a live image received from an image sensor and the representations of the at least two images superimposed onto the live image such that the representations indicate areas of a scene being depicted in the live image that have already been acquired and the live image also indicates areas of the scene that have not already been acquired;

providing, by one or more processors, the display with the composite image.

23. A method according to claim 22 wherein the position information comprises a 3-D rotation matrix that provides information about the roll, pitch, and yaw of the imaging apparatus.

24. A method according to claim 22 further comprising the step of providing a guide on the display to indicate whether sufficient images have been captured for generating the composite image.

25. A method according to claim 22 further comprising the step of providing touch controls to manipulate the representations.

26. A method according to claim 25 wherein the touch controls comprise interfaces for replacing or deleting one or all of the representations.

27. A method according to claim 25 wherein the touch controls act by raytracing from the 2-D location of a touch to a 3-D line representation in the space shown on the display.

28. A method according to claim 22 wherein SIFT is used to determine the overlap between the captured images.

29. A method according to claim 28 wherein the correction of position information is based on a correction factor generated by homography transformation of SIFT results.

30. A method according to claim 22 wherein the composite image is a panoramic image of the scene.