Image recognition in an augmented reality application

• US 8,810,599 B1
• Filed: 09/30/2011
• Issued: 08/19/2014
• Est. Priority Date: 11/02/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Tseng, Erick
• Primary Examiner(s): CHOW, JEFFREY J
• Application Number:
  US13/249,826
• Time in Patent Office: 1,054 Days
• Field of Search: 345/8, 345/158, 345/633, 345/581, 382/106, 382/154, 382/19, 340/435, 340/988, 340/539.13, 340/8.1, 701/300, 702/150, 702/5, 702/94, 702/85
• US Class Current: 345/633
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A computer-implemented augmented reality method includes obtaining an image acquired by a computing device running an augmented reality application, identifying image characterizing data in the obtained image, the data identifying characteristic points in the image, comparing the image characterizing data with image characterizing data for a plurality of geo-coded images stored by a computer server system, identifying locations of items in the obtained image using the comparison, and providing, for display on the computing device at the identified locations, data for textual or graphical annotations that correspond to each of the items in the obtained image, and formatted to be displayed with the obtained image or a subsequently acquired image.

Claims

1. A computer-implemented method, comprising:

obtaining, by a computer system, an image that was captured by a computing device;

identifying, by the computer system, image characterizing data for the obtained image, the image characterizing data identifying characteristic points of one or more items in the image;

obtaining, by the computer system, general device position information that indicates a geographic location at which the image was captured by the computing device and that was generated using information sensed by one or more position sensors of the computing device that sense geographic location information;

selecting, by the computer system, a plurality of geo-coded images based on the general device position information;

comparing the image characterizing data for the obtained image with image characterizing data for the plurality of geo-coded images;

determining, by the computer system and based on a result of comparing the image characterizing data for the obtained image with the image characterizing data for the plurality of geo-coded images, calculated device position information that indicates a refined geographic location at which the image was captured by the computing device; and

providing, by the computer system and to the computing device, calibration data for causing the computing device to calibrate, based on the calculated device position information, the one or more position sensors of the computing device that sense geographic location information.

2. The method of claim 1, further comprising:

providing, by the computer system and to the computing device, data for textual or graphical annotations to be displayed that correspond to at least some of the items in the obtained image, the data being formatted for display with the obtained image or a subsequently captured image;

identifying data that is indicative of interactions by a user of the computing device with the displayed textual or graphical annotations, and

using the data that is indicative of the interactions by the user of the computing device with the displayed textual or graphical annotations to compare subsequently captured images with the plurality of geo-coded images.

3. The method of claim 1, wherein determining the calculated device position information comprises determining a direction of aim of the computing device by computing an offset between a location at which one of the geo-coded images was captured and the general device position information that indicates the geographic location at which the image was captured and that was generated using information sensed by the one or more position sensors.

4. The method of claim 1, wherein the image characterizing data comprises data encoding corner points on the items in the images.

5. The method of claim 1, wherein determining the calculated device position information comprises computing outlines of the items in the obtained image.

6. The method of claim 5, further comprising identifying spatial locations within the identified outlines as anchor points for annotations that correspond to at least some of the items in the obtained image and providing, to the computing device, data that characterizes the identified spatial locations to cause the computing device to display the annotations with the obtained image on the computing device according to the identified spatial location.

7. The method of claim 1, further comprising:

obtaining, by the computer system along with the obtained image, range data that is generated based on information from one or more range sensors of the computing device and that indicates a distance of the one or more items in the obtained image from the computing device, and

using the range data to identify the geographic locations of the items in the obtained image.

8. The method of claim 1, further comprising selecting the plurality of geo-coded images from a larger group of geo-coded images using the general device position information obtained from the computing device.

9. The method of claim 8, wherein selecting the plurality of geo-coded images comprises selecting a sub-set of images that are coded as being acquired within a determined geographic distance from the geographic location indicated by the general device position information.

10. A computer-readable device storing instructions that when executed by a computer system cause the computer system to perform operations comprising:

obtaining an image that was captured by a computing device;

identifying image characterizing data for the obtained image, the image characterizing data identifying characteristic points of one or more items in the image;

obtaining general device position information that indicates a geographic location at which the image was captured by the computing device and that was generated using information sensed by one or more position sensors of the computing device that sense geographic location information;

selecting a plurality of geo-coded images based on the general device position information;

comparing the image characterizing data for the obtained image with image characterizing data for the plurality of geo-coded images;

determining, based on a result of comparing the image characterizing data for the obtained image with the image characterizing data for the plurality of geo-coded images, calculated device position information that indicates a refined geographic location at which the image was captured by the computing device; and

providing, to the computing device, calibration data for causing the computing device to calibrate, based on the calculated device position information, the one or more position sensors of the computing device that sense geographic location information.

11. The computer-readable device of claim 10, wherein the operations further comprise:

providing, to the computing device, data for textual or graphical annotations to be displayed that correspond to at least some of the items in the obtained image, the data being formatted for display with the obtained image or a subsequently captured image;

identifying data that is indicative of interactions by a user of the computing device with the displayed textual or graphical annotations, and

using the data that is indicative of the interactions by the user of the computing device with the displayed textual or graphical annotations to compare subsequently captured images with the plurality of geo-coded images.

12. The computer-readable device of claim 10, wherein determining the calculated device position information comprises determining a direction of aim of the computing device by computing an offset between a location at which one of the geo-coded images was captured and the general device position information that indicates the geographic location at which the image was captured and that was generated using information sensed by the one or more position sensors.

13. The computer-readable device of claim 10, wherein the image characterizing data comprises data encoding corner points on the items in the images.

14. The computer-readable device of claim 10, wherein determining the calculated device position information comprises computing outlines of the items in the obtained image.

15. The computer-readable device of claim 14, wherein the operations further comprise identifying spatial locations within the identified outlines as anchor points for annotations that correspond to at least some of the items in the obtained image and providing, to the computing device, data that characterizes the identified spatial locations to cause the computing device to display the annotations with the obtained image on the computing device according to the identified spatial location.

16. The computer-readable device of claim 10, wherein the operations further comprise:

obtaining, along with the obtained image, range data that is generated based on information from one or more range sensors of the computing device and that indicates a distance of the one or more items in the obtained image from the computing device, and

using the range data to identify the geographic locations of the items in the obtained image.

17. The computer-readable device of claim 10, wherein the operations further comprise selecting the plurality of geo-coded images from a larger group of geo-coded images using the general device position information obtained from the computing device.

18. The computer-readable device of claim 17, wherein selecting the plurality of geo-coded images comprises selecting a sub-set of images that are coded as being acquired within a determined geographic distance from the geographic location indicated by the general device position information.

19. A computer system, comprising:

one or more processors,

a computer-readable medium storing instructions that when executed by the one or more processors cause the computing system to perform operations comprising;

obtaining an image that was captured by a computing device;

identifying image characterizing data for the obtained image, the image characterizing data identifying characteristic points of one or more items in the image;

obtaining general device position information that indicates a geographic location at which the image was captured by the computing device and that was generated using information sensed by one or more position sensors of the computing device that sense geographic location information;

selecting a plurality of geo-coded images based on the general device position information;

comparing the image characterizing data for the obtained image with image characterizing data for the plurality of geo-coded images;

determining, based on a result of comparing the image characterizing data for the obtained image with the image characterizing data for the plurality of geo-coded images, calculated device position information that indicates a refined geographic location at which the image was captured by the computing device; and

providing, to the computing device, calibration data for causing the computing device to calibrate, based on the calculated device position information, the one or more position sensors of the computing device that sense geographic location information.

20. The computer system of claim 19, wherein the operations further comprise:

providing, to the computing device, data for textual or graphical annotations to be displayed that correspond to at least some of the items in the obtained image, the data being formatted for display with the obtained image or a subsequently captured image;

identifying data that is indicative of interactions by a user of the computing device with the displayed textual or graphical annotations, and

using the data that is indicative of the interactions by the user of the computing device with the displayed textual or graphical annotations to compare subsequently captured images with the plurality of geo-coded images.

21. The computer system of claim 19, wherein determining the calculated device position information comprises determining a direction of aim of the computing device by computing an offset between a location at which one of the geo-coded images was captured and the general device position information that indicates the geographic location at which the image was captured and that was generated using information sensed by the one or more position sensors.

22. The computer system of claim 19, wherein the image characterizing data comprises data encoding corner points on the items in the images.

23. The computer system of claim 19, wherein determining the calculated device position information comprises computing outlines of the items in the obtained image.

24. The computer system of claim 23, wherein the operations further comprise identifying spatial locations within the identified outlines as anchor points for annotations that correspond to at least some of the items in the obtained image and providing, to the computing device, data that characterizes the identified spatial locations to cause the computing device to display the annotations with the obtained image on the computing device according to the identified spatial location.

25. The computer system of claim 19, wherein the operations further comprise:

obtaining, along with the obtained image, range data that is generated based on information from one or more range sensors of the computing device and that indicates a distance of the one or more items in the obtained image from the computing device, and

using the range data to identify the geographic locations of the items in the obtained image.

26. The computer system of claim 19, wherein the operations further comprise selecting the plurality of geo-coded images from a larger group of geo-coded images using the general device position information obtained from the computing device.

27. The computer system of claim 26, wherein selecting the plurality of geo-coded images comprises selecting a sub-set of images that are coded as being acquired within a determined geographic distance from the geographic location indicated by the general device position information.