Graphical refinement for points of interest

• US 9,269,011 B1
• Filed: 02/11/2013
• Issued: 02/23/2016
• Est. Priority Date: 02/11/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sikka, Avnish, Jahagirdar, Sonjeev, Sassano, James, Wu, Pengcheng, Sovich, Nicholas Randal
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Gilliard, Delomia L
• Application Number:
  US13/764,646
• Time in Patent Office: 1,107 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

Various embodiments crowd source images to cover various angles, zoom levels, and elevations of objects and/or points of interest (POIs) while under various lighting conditions. The crowd sourced images are tagged or associated with a particular POI or geographic location and stored in a database for use by an augmented reality (AR) application to recognize objects appearing in a live view of a scene captured by at least one camera of a computing device. The more comprehensive the database, the more accurately an object or POI in the scene will be recognized and/or tracked by the AR application. Accordingly, the more accurately an object is recognized and tracked by the AR application, the more smoothly and continuous the content and movement transitions thereof can be presented to users in the live view.

Claims

1. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

receiving a first image and sensor data associated with the first image of a point of interest (POI) from a first user of a first computing device;

recognizing the POI in the first image by comparing the first image to images stored for POIs within a predetermined distance of the first computing device;

aggregating the first image with related images stored for the POI using the associated sensor data of the first image to determine a first view of the POI in the first image;

receiving a second image and sensor data associated with the second image from a second user of a second computing device;

recognizing the POI in the second image by comparing the second image to the images stored for the POIs within the predetermined distance of the second computing device;

aggregating the second image with the first image and the related images stored for the POI using the associated sensor data of the second image to determine a second view of the POI in the second image; and

based at least in part on the first image and the second image, refining placement of an augmented reality banner for the POI displayed in an overlay of a live view being captured with at least one camera of a third computing device, wherein the augmented reality banner includes at least name information of the POI;

receiving feedback data indicating that the augmented reality banner for the POI is incorrectly named;

determining that the feedback data meets a threshold accuracy level to generate a corrected augmented reality banner; and

presenting the corrected augmented reality banner for the POI based at least in part on the feedback data.

2. The computer-implemented method of claim 1, wherein recognizing the POI includes using at least one of a scale-invariant feature transform (SIFT) or speeded up robust feature (SURF) algorithm.

3. The computer-implemented method of claim 1, further comprising:

enabling subsequent users to provide additional images of the POI to further refine placement of the augmented reality banner in the overlay.

4. The computer-implemented method of claim 1, wherein the first image and the second image include location tags for the POI provided by the first user and the second user.

5. The computer-implemented method of claim 1, wherein the sensor data is data from at least one of a global positioning system (GPS) sensor, an electronic compass, a gyroscope, or an accelerometer.

6. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

receiving at least one image of a point of interest (POI) from a user of a computing device;

recognizing the POI in the at least one image;

aggregating the at least one image with related images stored for the POI;

based at least in part on a location of the POI in the at least one received image and the related images stored for a respective view of the POI, displaying an augmented reality banner for the POI in an overlay of a live view being captured with at least one camera of the computing device, the augmented reality banner includes at least name information of the POI;

receiving feedback data indicating that the augmented reality banner for the POI is incorrectly named;

determining that the feedback data meets a threshold accuracy level to generate a corrected augmented reality banner; and

presenting the corrected augmented reality banner for the POI based at least in part on the feedback data.

7. The computer-implemented method of claim 6, further comprising:

recognizing text in the at least one image associated with at least one sign using at least one of an optical character recognition engine or logo detection algorithm.

8. The computer-implemented method of claim 6, wherein receiving at least one image of the POI includes sensor data associated with the at least one image for at least determining the respective view of the POI.

9. The computer-implemented method of claim 8, further comprising:

based at least in part on the sensor data associated with the at least one image, determining a view of the POI in the at least one image; and

stitching the at least one image together with the related images stored for the respective view of the POI.

10. The computer-implemented method of claim 9, wherein the sensor data is data from at least one of a global positioning system (GPS) sensor, an electronic compass, a gyroscope, or an accelerometer.

11. The computer-implemented method of claim 6, further comprising:

preloading images of POI within a predetermined distance of the computing device, wherein recognizing the POI includes comparing the at least one image to the preloaded images of POI.

12. The computer-implemented method of claim 6, further comprising:

enabling users to provide feedback to correct a tag for an incorrectly tagged POI; and

correcting the tag for the POI upon receiving a threshold amount of feedback.

13. The computer-implemented method of claim 12, wherein recognizing the POI includes using at least one of a scale-invariant feature transform (SIFT) or speeded up robust feature (SURF) algorithm.

14. The computer-implemented method of claim 6, further comprising:

enabling subsequent users to provide additional images of the POI to further refine placement of the augmented reality banner in the overlay.

15. The computer-implemented method of claim 12, further comprising:

enabling users to tag an image with a geographic location of the POI.

16. A computing device, comprising:

a processor;

a display screen; and

memory including instructions that, when executed by the processor, cause the computing device to;

receive at least one image of a point of interest (POI) from a user of a computing device;

recognize the POI in the at least one image;

aggregate the at least one image with related images stored for the POI;

based at least in part on a location of the POI in the at least one received image and the related images stored for a respective view of the POI, display an augmented reality banner for the POI in an overlay of a live view being captured with at least one camera of the computing device, the augmented reality banner includes at least name information of the POI;

receive feedback data indicating that the augmented reality banner for the POI is incorrectly named;

determine that the feedback data meets a threshold accuracy level to generate a corrected augmented reality banner; and

present the corrected augmented reality banner for the POI based at least in part on the feedback data.

17. The computing device of claim 16, wherein the instructions, when executed by the processor, further enable the computing device to:

based at least in part on the sensor data associated with the at least one image, determine a view of the POI in the at least one image; and

stitch the at least one image together with the related images stored for the respective view of the POI.

18. The computing device of claim 16, wherein the augmented reality tag is displayed in the overlay of the live view in response to a visual search query by the user.

19. The computing device of claim 16, wherein the instructions, when executed by the processor, further enable the computing device to:

enable subsequent users to provide additional images of the POI to further refine placement of the augmented reality banner in the overlay.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause a computing device to:

receive at least one image of a point of interest (POI) from a user of a computing device;

recognize the POI in the at least one image;

aggregate the at least one image with related images stored for the POI;

based at least in part on a location of the POI in the at least one received image and the related images stored for a respective view of the POI, display an augmented reality banner for the POI in an overlay of a live view being captured with at least one camera of the computing device, the augmented reality banner includes at least name information of the POI;

receive feedback data indicating that the augmented reality banner for the POI is incorrectly named;

determine that the feedback data meets a threshold accuracy level to generate a corrected augmented reality banner; and

present the corrected augmented reality banner for the POI based at least in part on the feedback data.

21. The non-transitory computer-readable storage medium of claim 20, wherein the augmented reality tag is displayed in the overlay of the live view in response to a visual search query by the user.

22. The non-transitory computer-readable storage medium of claim 20, wherein aggregating the images includes:

based at least in part on the sensor data associated with the at least one image, determine a view of the POI in the at least one image; and

stitch the at least one image together with the related images stored for the respective view of the POI.

23. The non-transitory computer-readable storage medium of claim 20,

preload images of POI within a predetermined distance of the computing device.

24. The non-transitory computer-readable storage medium of claim 23, wherein recognizing the POI includes comparing the at least one image to the preloaded images of POI.

25. The non-transitory computer-readable storage medium of claim 24, wherein recognizing the POI includes using at least one of a scale-invariant feature transform (SIFT) or speeded up robust feature (SURF) algorithm.