Image recognition in an augmented reality application

• US 8,890,896 B1
• Filed: 11/02/2010
• Issued: 11/18/2014
• Est. Priority Date: 11/02/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Tseng, Erick
• Primary Examiner(s): CHOW, JEFFREY J
• Application Number:
  US12/938,132
• Time in Patent Office: 1,477 Days
• Field of Search: 345/8, 345/158, 345/633, 382/106, 382/154
• US Class Current: 345/633
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A computer-implemented augmented reality method includes obtaining an image acquired by a computing device running an augmented reality application, identifying image characterizing data in the obtained image, the data identifying characteristic points in the image, comparing the image characterizing data with image characterizing data for a plurality of geo-coded images stored by a computer server system, identifying locations of items in the obtained image using the comparison, and providing, for display on the computing device at the identified locations, data for textual or graphical annotations that correspond to each of the items in the obtained image, and formatted to be displayed with the obtained image or a subsequently acquired image.

Claims

1. A computer-implemented method, comprising:

obtaining, by one or more processors, an image;

identifying, by the one or more processors, image characterizing data for the obtained image, the image characterizing data identifying characteristic points in the obtained image;

comparing, by the one or more processors, the image characterizing data for the obtained image with image characterizing data for a plurality of geo-coded images;

identifying, by the one or more processors, one or more items in the obtained image based on the comparison of the image characterizing data for the obtained image with the image characterizing data for the plurality of geo-coded images;

determining, by the one or more processors, geographic locations of the one or more items in the obtained image using predetermined geographic location information for the one or more items;

identifying, by the one or more processors and based on the image characterizing data for the obtained image that identifies characteristic points in the obtained image, an additional item in the obtained image for which predetermined geographic location information is not available; and

determining, by the one or more processors, a geographic location of the additional item in the obtained image for which predetermined geographic location information is not available based at least on

(i) spatial relationship in the obtained image between a representation of the additional item in the obtained image and respective representations of the one or more items in the obtained image, and

(ii) the determined geographic locations of the one or more items in the obtained image.

2. The method of claim 1, further comprising:

providing, to a computing device, data for textual or graphical annotations that correspond to the one or more items in the obtained image, the data formatted to be displayed with the obtained image or a subsequently acquired image;

identifying data that is indicative of interactions by a user of the computing device with the textual or graphical annotations displayed on the computing device, and

storing the data with the obtained image for comparison to subsequently obtained images, and with the plurality of geo-coded images.

3. The method of claim 1, wherein the image characterizing data comprises data encoding corner points on items in the images.

4. The method of claim 1, wherein determining geographic locations of the one or more items in the obtained image comprises computing outlines of the one or more items in the obtained image.

5. The method of claim 4, further comprising identifying spatial locations within the identified outlines as anchor points for textual or graphical annotations that correspond to the one or more items in the obtained image and that are to be displayed on a computing device.

6. The method of claim 1, further comprising:

obtaining, by the one or more processors along with the obtained image, range data that is generated based on information from one or more range sensors of a computing device and that indicates a distance of at least one item in the obtained image from the computing device, and

using the range data to determine the geographic locations of the one or more items in the obtained image.

7. The method of claim 1, wherein the geographic location of the additional item in the obtained image cannot be determined based on comparing image characterizing data of the additional item in the obtained image with the image characterizing data for the plurality of geo-coded images.

8. The method of claim 1, wherein the spatial relationship in the obtained image and the determined geographic locations of the one or more items in the obtained image are used in determining the geographic location of the additional item in the obtained image in response to determining that predetermined geographic location information is not available for the additional item.

9. The method of claim 1, wherein determining the geographic location of the additional item in the obtained image is further based on at least one of a geographic location at which the obtained image was captured, a viewing direction at which the obtained image was captured, and a vertical viewing angle at which the obtained image was captured.

10. A computer-readable device storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

obtaining an image acquired by a computing device;

identifying image characterizing data for the obtained image, the image characterizing data identifying characteristic points in the obtained image;

comparing the image characterizing data for the obtained image with image characterizing data for a plurality of geo-coded images;

identifying one or more items in the obtained image using a result of the comparison of the image characterizing data for the obtained image with the image characterizing data for the plurality of geo-coded images;

determining geographic locations of the one or more items in the obtained image using predetermined geographic location information for the one or more items;

identifying, based on the image characterizing data for the obtained image that identifies characteristic points in the obtained image, an additional item in the obtained image for which predetermined geographic location information is not available; and

determining a geographic location of the additional item in the obtained image for which predetermined geographic location information is not available based at least on

(i) a spatial relationship in the obtained image between a representation of the additional item in the obtained image and respective representations of the one or more items in the obtained image, and

(ii) the determined geographic locations of the one or more items in the obtained image.

11. The computer-readable device of claim 10, wherein the operations further comprise:

providing, to the computing device, data for textual or graphical annotations that correspond to the one or more items in the obtained image, the data formatted to be displayed with the obtained image or a subsequently acquired image;

identifying data that is indicative of interactions by a user of the computing device with the textual or graphical annotations displayed on the computing device, and

storing the data with the obtained image for comparison to subsequently obtained images, and with the plurality of geo-coded images.

12. The computer-readable device of claim 10, wherein the image characterizing data comprises data encoding corner points on items in the images.

13. The computer-readable device of claim 10, wherein determining geographic locations of the one or more items in the obtained image comprises computing outlines of the one or more items in the obtained image.

14. The computer-readable device of claim 13, wherein the operations further comprise identifying spatial locations within the identified outlines as anchor points for textual or graphical annotations that correspond to the one or more items in the obtained image and that are to be displayed on the computing device.

15. The computer-readable device of claim 10, wherein the operations further comprise:

obtaining, along with the obtained image, range data that is generated based on information from one or more range sensors of the computing device and that indicates a distance of at least one item in the obtained image from the computing device, and

using the range data to determine the geographic locations of the one or more items in the obtained image.

16. The computer-readable device of claim 10, wherein the geographic location of the additional item in the obtained image cannot be determined based on comparing image characterizing data of the additional item in the obtained image with the image characterizing data for the plurality of geo-coded images.

17. A system comprising:

one or more processors; and

one or more computer-readable media having instructions stored thereon that, when executed by the one or more processors, cause performance of operations comprising;

obtaining an image acquired by a computing device;

identifying image characterizing data for the obtained image, the image characterizing data identifying characteristic points in the obtained image;

comparing the image characterizing data for the obtained image with image characterizing data for a plurality of geo-coded images;

identifying one or more items in the obtained image using a result of the comparison of the image characterizing data for the obtained image with the image characterizing data for the plurality of geo-coded images;

determining geographic locations of the one or more items in the obtained image using predetermined geographic location information for the one or more items;

identifying, based on the image characterizing data for the obtained image that identifies characteristic points in the obtained image, an additional item in the obtained image for which predetermined geographic location information is not available; and

determining a geographic location of the additional item in the obtained image for which predetermined geographic location information is not available based at least on

(i) a spatial relationship in the obtained image between a representation of the additional item in the obtained image and respective representations of the one or more items in the obtained image, and

(ii) the determined geographic locations of the one or more items in the obtained image.

18. The system of claim 17, wherein the operations further comprise:

providing, to the computing device, data for textual or graphical annotations that correspond to the one or more items in the obtained image, the data formatted to be displayed with the obtained image or a subsequently obtained image;

identifying data that is indicative of interactions by a user of the computing device with the textual or graphical annotations displayed on the computing device, and

storing the data with the obtained image for comparison to subsequently obtained images, and with the plurality of geo-coded images.

19. The system of claim 17, wherein the image characterizing data comprises data encoding corner points on items in the images.

20. The system of claim 17, wherein the operations further comprise:

obtaining, along with the obtained image, range data that is generated based on information from one or more range sensors of the computing device and that indicates a distance of at least one item in the obtained image from the computing device, and

using the range data to determine the geographic locations of the one or more items in the obtained image.

21. The system of claim 17, wherein the geographic location of the additional item in the obtained image cannot be determined based on comparing image characterizing data of the additional item in the obtained image with the image characterizing data for the plurality of geo-coded images.