Matching An Approximately Located Query Image Against A Reference Image Set

• US 20110135207A1
• Filed: 12/07/2009
• Published: 06/09/2011
• Est. Priority Date: 12/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Buddemeier, Ulrich, Brucher, Fernando, Neven, Hartmut, Stewenius, Henrik, Adam, Hartwig, Flynn, John
• Granted Patent: US 8,189,964 B2
• Time in Patent Office: Days
• US Class Current: 382/209
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/39 Urban scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Aspects of the invention pertain to matching a selected image/photograph against a database of reference images having location information. The image of interest may include some location information itself, such as latitude/longitude coordinates and orientation. However, the location information provided by a user'"'"'s device may be inaccurate or incomplete. The image of interest is provided to a front end server, which selects one or more cells to match the image against. Each cell may have multiple images and an index. One or more cell match servers compare the image against specific cells based on information provided by the front end server. An index storage server maintains index data for the cells and provides them to the cell match servers. If a match is found, the front end server identifies the correct location and orientation of the received image, and may correct errors in an estimated location of the user device.

Claims

1. An image processing method, comprising:

receiving an image request from a user device, the image request including an image of interest and location metadata for the image of interest;

analyzing the location metadata to select one or more cells to evaluate against the image of interest, each cell having one or more geolocated images and index data associated therewith;

for each selected cell, comparing the image of interest against the index data of that cell;

identifying any matches from the geolocated images of the selected cells based on the compared index data; and

providing the matches.

2. The image processing method of claim 1, wherein the matches are provided along with a match confidence indicator that identifies a likelihood or accuracy of each match.

3. The image processing method of claim 2, wherein a value of the match confidence indicator depends on geolocation verification between the location metadata and location information for the geolocated images of the selected cells.

4. The image processing method of claim 1, wherein updated location metadata for the image of interest is provided to the user device along with the matches.

5. The image processing method of claim 1, wherein the index data is stored in an index storage server, and the index data for each selected cell is accessed with a key representing that cell'"'"'s unique ID.

6. The image processing method of claim 1, wherein the index data corresponds to features of the geolocated images.

7. the image processing method of claim 6, wherein the features are selected from the set consisting of corners, edges or lines, brightness information and histogram information.

8. The image processing method of claim 6, wherein the geolocated images are stored in an image database and the index data is stored in a cell database.

9. The image processing method of claim 6, wherein the index data is stored in a k-dimensional tree format.

10. The image processing method of claim 1, wherein each cell has a unique ID derived from geolocation coordinates of that cell.

11. An image processing apparatus, comprising:

a front end module configured to receive an image request from a user device, the image request including an image of interest and location metadata for the image of interest, the front end module being further configured to analyze the location metadata to select one or more cells to evaluate against the image of interest, each cell having one or more geolocated images and index data associated therewith; and

a cell match module configured to compare the image of interest against the index data of the selected cells and to identify any matches from the geolocated images of the selected cells based on the compared index data.

12. The image processing apparatus of claim 11, wherein the cell match module comprises a plurality of cell match servers, and given ones of the cell match servers are assigned to perform the comparison for a corresponding one of the selected cells.

13. The image processing apparatus of claim 12, wherein the matches are provided along with a match confidence indicator that identifies a likelihood or accuracy of each match.

14. The image processing apparatus of claim 12, further comprising an indexed module configured to store the index data of each cell.

15. The image processing apparatus of claim 14, wherein each cell has a unique ID associated therewith, and each given cell match server accesses the index data of the corresponding cell from the indexed module using a key representing the unique ID of that cell.

16. The image processing apparatus of claim 15, wherein the unique ID for each cell is derived from geolocation coordinates of that cell.

17. The image processing apparatus of claim 11, wherein the index data corresponds to features of the geolocated images.

18. The image processing apparatus of claim 17, wherein the features are selected from the set consisting of corners, edges or lines, brightness information and histogram information.