Apparatus and method for feature matching in distorted images

• US 8,818,101 B1
• Filed: 01/03/2012
• Issued: 08/26/2014
• Est. Priority Date: 01/03/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lim, Jongwoo
• Primary Examiner(s): Lu, Tom Y
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US13/342,644
• Time in Patent Office: 966 Days
• Field of Search: None
• US Class Current: 382/190
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G09G 2356/00 Detection of the display position w.r.t. other display screens
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data

Abstract

Provided is a method of processing images. The method may include obtaining a plurality of radially-distorted images captured from one location in different directions, the plurality of images each having a field of view, and neighboring images among the plurality of images having overlapping fields of view. The method may further include reducing relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images, matching features between the reduced distortion portions of neighboring images in the overlapping fields of view, and storing in memory data indicative of matched features.

Claims

1. A method, comprising:

obtaining, by one or more computing devices, a plurality of radially-distorted images captured from one location in different directions, the plurality of images each having a field of view, and neighboring images among the plurality of images having overlapping fields of view;

reducing, by the one or more computing devices, relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images by warping the portions of the neighboring images in the overlapping fields of view of the neighboring images by barrel transforming the portions of the neighboring images in the overlapping fields of view of the neighboring images, the barrel transforms being centered beyond the field of view of the neighboring images in opposite directions for each of the neighboring images;

matching, by the one or more computing devices, features between the reduced distortion portions of neighboring images in the overlapping fields of view; and

storing, by the one or more computing devices, in memory data indicative of the matched features.

2. The method of claim 1 wherein warping the portions of the neighboring images in the overlapping fields of view of the neighboring images comprises:

warping the portions of the neighboring images in the overlapping fields of view of the neighboring images based on a fisheye lens model.

3. The method of claim 1 wherein reducing relative distortion of portions of neighboring images in the overlapping fields of view comprises:

applying a first transform configured to reduce distortion in a left portion of one image among the neighboring images; and

applying a second transform configured to reduce distortion in a right portion of another image among the neighboring images.

4. The method of claim 1 wherein matching features between the reduced distortion portions of neighboring images in the overlapping fields of view comprises:

detecting corners of edges in the reduced distortion portions of neighboring images in the overlapping fields of view; and

comparing corners of edges detected in the reduced distortion portions of one neighboring image with corners of edges detected in the reduced distortion portions of another neighboring image next to the one neighboring image.

5. The method of claim 4 wherein detecting corners of edges in the reduced distortion portions of neighboring images in the overlapping fields of view comprises:

determining image gradients of pixels in the reduced distortion portions of neighboring images; and

determining whether the image gradients are greater than a gradient threshold.

6. The method of claim 5 wherein the image gradients are first order image gradients.

7. The method of claim 5 wherein the image gradients are second order image gradients.

8. The method of claim 1 wherein reducing relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images comprises reducing relative distortion of non-overlapping portions of the neighboring images.

9. The method of claim 1 wherein reducing relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images comprises reducing distortion of neighboring images relative to a rectilinear representation of the neighboring images.

10. The method of claim 1 wherein reducing relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images comprises inverting the positions of pixels of one of the neighboring images.

11. The method of claim 1 wherein reducing relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images comprises transforming each of a plurality of trapezium-shaped regions of the neighboring images to a rectangular-shaped region.

12. The method claim 1 comprising:

forming a panorama by stitching portions of the neighboring images to one another based on the data indicative of matched features; and

storing the panorama in memory.

13. The method of claim 12 wherein forming a panorama by stitching portions of the neighboring images to one another based on the data indicative of matched features comprises:

positioning the neighboring images relative to one another by aligning matching features in the neighboring images with one another.

14. The method of claim 12 comprising:

receiving a request from a computing device for at least a portion of the panorama; and

retrieving the requested at least a portion of the panorama from memory; and

transmitting the retrieved at least a portion of the panorama to the computing device.

15. The method of claim 14 comprising:

determining at least one of the following;

whether a user associated with the computing device has subscribed to a panorama-providing service;

whether a user associated with the computing device has a license to view images from the panorama-providing service;

or an advertisement to be transmitted to the user based on the requested at least a portion of the panorama.

16. The method of claim 1 wherein warping the portions of the neighboring images in the overlapping fields of view of the neighboring images comprises:

warping the portions of the neighboring images in the overlapping fields of view of the neighboring images based on a fisheye lens model;

wherein matching features between the reduced distortion portions of neighboring images in the overlapping fields of view comprises;

detecting edges in the reduced distortion portions of neighboring images in the overlapping fields of view based on image gradients of pixels in the reduced distortion portions of neighboring images; and

comparing edges detected in the reduced distortion portions of one neighboring image with edges detected in the reduced distortion portions of another neighboring image next to the one neighboring image.

17. A non-transitory tangible machine-readable medium having a computer program product stored thereon that causes a computer system to effect a process comprising the following:

obtaining a plurality of radially-distorted images captured from one location in different directions, the plurality of images each having a field of view, and neighboring images among the plurality of images having overlapping fields of view;

reducing relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images by warping the portions of the neighboring images in the overlapping fields of view of the neighboring images by barrel transforming the portions of the neighboring images in the overlapping fields of view of the neighboring images, the barrel transforms being centered beyond the field of view of the neighboring images in opposite directions for each of the neighboring images;

matching features between the reduced distortion portions of neighboring images in the overlapping fields of view; and

storing in memory data indicative of the matched features.

18. A system, comprising:

one or more processors;

one or more memories communicatively coupled to the one or more processors and storing instructions that when executed by one or more of the one or more processors cause a process comprising the following to occur;

obtaining a plurality of radially-distorted images captured from one location in different directions, the plurality of images each having a field of view, and neighboring images among the plurality of images having overlapping fields of view;

reducing relative distortion of portions of the neighboring images in the overlapping fields of view of the neighboring images by warping the portions of the neighboring images in the overlapping fields of view of the neighboring images by barrel transforming the portions of the neighboring images in the overlapping fields of view of the neighboring images, the barrel transforms being centered beyond the field of view of the neighboring images in opposite directions for each of the neighboring images;

matching features between the reduced distortion portions of neighboring images in the overlapping fields of view; and

storing in memory data indicative of the matched features.

19. The system of claim 18 wherein matching features between the reduced distortion portions of neighboring images in the overlapping fields of view comprises:

detecting corners of edges in the reduced distortion portions of neighboring images in the overlapping fields of view; and

comparing corners of edges detected in the reduced distortion portions of one neighboring image with corners of edges detected in the reduced distortion portions of another neighboring image next to the one neighboring image.

20. The system of claim 19 wherein detecting corners of edges in the reduced distortion portions of neighboring images in the overlapping fields of view comprises:

determining image gradients of pixels in the reduced distortion portions of neighboring images; and

determining whether the image gradients are greater than a gradient threshold.