Providing information in an image-based information retrieval system

• US 8,315,423 B1
• Filed: 12/28/2007
• Issued: 11/20/2012
• Est. Priority Date: 12/28/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Jing, Yushi, Baluja, Shumeet
• Primary Examiner(s): Liew, Alex
• Application Number:
  US11/966,471
• Time in Patent Office: 1,789 Days
• Field of Search: 382/305, 382/306
• US Class Current: 382/100
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices

Abstract

Techniques are described for providing information in an image-based information retrieval system. An image including an object is received from a mobile device over a network of computer. The object included the image is matched with a stored representation of the object. Information related to the object is identified based on an association between the identified information and the stored representation of the object. Presentation, over the network of computers on the mobile device, of the identified information is enabled.

Claims

1. A method for providing information comprising:

receiving, from a mobile device capable of generating digital photographs over a network of computers, a digital photograph obtained by the mobile device including an object for which an operator of the mobile telephone desires information;

receiving, from the mobile device over a network, location information from a global positioning system associated with the mobile device;

using object recognition techniques to match the object included in the digital photograph with a stored representation of the object;

identifying non-advertisement content related to the object based on an association between the identified non-advertisement content and the stored representation of the object;

identifying one or more advertisements related to the object; and

sending, over the network of computers to the mobile device, the identified non-advertisement content and one or more advertisements for presentation;

wherein identifying the non-advertisement content related to the object comprises identifying the non-advertisement content based on the association between the identified information and the stored representation of the object and the location information.

2. The method of claim 1 wherein the object comprises one of a product for sale, an indication of a name of a restaurant, an indication of a name of a hotel, and an indication of an entertainment venue.

3. The method of claim 1 wherein the mobile device comprises one of a mobile telephone, a mobile electronic messaging device, a personal digital assistant, a music player, a gaming device, a video player and a navigational device.

4. The method of claim 1 wherein the mobile device is configured to generate the digital photograph using a camera included in the mobile device.

5. The method of claim 1 wherein:

the mobile device comprises a digital camera, and

the digital photograph comprises a digital photograph of a restaurant.

6. The method of claim 1 wherein using object recognition techniques to match the object included the image with a stored representation of the object comprises using one or more of an optical character recognition engine, a rigid textured object recognition engine, a face recognition engine, and an articulate object engine.

7. The method of claim 6, wherein matching the object comprises:

comparing one or more stored representations of an object with the object using the engines to generate a confidence value; and

selecting the stored representation of an object with the highest confidence value as a match.

8. The method of claim 1 wherein:

receiving the digital photograph comprises receiving a digital photograph of an item from a retail store that is also available from an online merchant;

identifying the non-advertisement content related to the object comprises identifying search engine results related to the item; and

identifying the one or more advertisements related to the object comprises identifying an advertisement from the online merchant related to the item.

9. A computer system comprising instructions, that when executed by the computer system, cause the computer system to perform operations comprising:

receiving, from a mobile device capable of generating digital photographs over a network of computers, a digital photograph obtained by the mobile device including an object;

receiving, from the mobile device over a network, location information from a global positioning system associated with the mobile device

using object recognition techniques to match the object included in the digital photograph with a stored representation of the object;

identifying non-advertisement content related to the object based on an association between the identified non-advertisement content and the stored representation of the object and the location information;

identifying one or more advertisements related to the object; and

enabling presentation, over the network of computers on the mobile device, the identified non-advertisement content and one or more advertisements.

10. The computer system of claim 9 wherein the mobile device comprises one of a mobile telephone, a mobile electronic messaging device, a personal digital assistant, a music player, a gaming device, a video player and a navigational device.

11. The computer system of claim 9 wherein the mobile device is configured to generate a digital image.

12. The computer system of claim 9 wherein:

the mobile device comprises a digital camera, and

the digital photograph comprises a digital photograph of a restaurant.

13. The computer system of claim 9 wherein using object recognition techniques to match the object including the image with a stored representation of the object comprises using one of a optical character recognition engine, a rigid textured object recognition engine, a face recognition engine, and an articulate object engine.

14. The computer system of claim 13, wherein using object recognition techniques to match the object comprises:

comparing one or more stored representations of an object with the object using the engines to generate a confidence value; and

selecting the stored representation of an object with the highest confidence value as a match.

15. The computer system of claim 9 wherein enabling presentation comprises enabling presentation of information including both sponsored-content with content other than sponsored content.

16. A computer system comprising:

one or more processors configured to receive from a mobile device capable of generating digital photographs over a network of computers, an image obtained by the mobile device including an object and network location information from a global positioning system associated with the mobile device;

one or more object recognition units configured to use object recognition techniques to match the object included in the digital photograph with a stored representation of the object;

means for identifying non-advertisement content related to the object based on an association between the identified non-advertisement content and the stored representation of the object and the location information;

means for identifying one or more advertisements related to the object; and

a presentation device configured to enable presentation, over the network of computers on the mobile device, the identified non-advertisement content and one or more advertisements.

17. A method for providing information comprising:

receiving, from a mobile device capable of generating digital photographs over a network of computers, a digital photograph obtained by the mobile device including an object for which an operator of the mobile telephone desires information;

using object recognition techniques to match the object included in the digital photograph with a stored representation of the object;

identifying non-advertisement content related to the object based on an association between the identified non-advertisement content and the stored representation of the object;

identifying one or more advertisements related to the object; and

sending, over the network of computers to the mobile device, the identified non-advertisement content and one or more advertisements for presentation, wherein

receiving the digital photograph comprises receiving a digital photograph of an item from a retail store that is also available from an online merchant;

identifying the non-advertisement content related to the object comprises identifying search engine results related to the item; and

identifying the one or more advertisements related to the object comprises identifying an advertisement from the online merchant related to the item.

18. The method of claim 17 wherein the object comprises one of a product for sale, an indication of a name of a restaurant, an indication of a name of a hotel, and an indication of an entertainment venue.

19. The method of claim 17, further comprising:

receiving, from the mobile device over the network location information from a global positioning system associated with the mobile device;

wherein identifying the non-advertisement content related to the object comprises identifying the non-advertisement content based on the association between the identified information and the stored representation of the object and the location information.

20. The method of claim 17 wherein the mobile device comprises one of a mobile telephone, a mobile electronic messaging device, a personal digital assistant, a music player, a gaming device, a video player and a navigational device.

21. The method of claim 17 wherein the mobile device is configured to generate the digital photograph using a camera included in the mobile device.

22. The method of claim 17 wherein:

the mobile device comprises a digital camera, and

the digital photograph comprises a digital photograph of a restaurant.

23. The method of claim 17 wherein using object recognition techniques to match the object included the image with a stored representation of the object comprises using one or more of an optical character recognition engine, a rigid textured object recognition engine, a face recognition engine, and an articulate object engine.

24. The method of claim 23, wherein matching the object comprises:

comparing one or more stored representations of an object with the object using the engines to generate a confidence value; and

selecting the stored representation of an object with the highest confidence value as a match.

25. A computer system comprising instructions, that when executed by the computer system, cause the computer system to perform operations comprising:

receiving, from a mobile device'"'"'capable of generating digital photographs over a network of computers, a digital photograph obtained by the mobile device including an object, the object comprising an item from a retail store that is also available from an online merchant;

using object recognition techniques to match the object included in the digital photograph with a stored representation of the object;

identifying non-advertisement content related to the object based on an association between the identified non-advertisement content and the stored representation of the object, the non-advertisement content including search engine results related to the item;

identifying one or more advertisements related to the object including an advertisement from the online merchant related to the item; and

enabling presentation, over the network of computers on the mobile device, the identified non-advertisement content and one or more advertisements.

26. The computer system of claim 25 wherein the mobile device comprises one of a mobile telephone, a mobile electronic messaging device, a personal digital assistant, a music player, a gaming device, a video player and a navigational device.

27. The computer system of claim 25 wherein the mobile device is configured to generate a digital image.

28. The computer system of claim 25 wherein:

the mobile device comprises a digital camera, and

the digital photograph comprises a digital photograph of a restaurant.

29. The computer system of claim 25 wherein using object recognition techniques to match the object including the image with a stored representation of the object comprises using one of a optical character recognition engine, a rigid textured object recognition engine, a face recognition engine, and an articulate object engine.

30. The computer system of claim 29, wherein using object recognition techniques to match the object comprises:

comparing one or more stored representations of an object with the object using the engines to generate a confidence value; and

selecting the stored representation of an object with the highest confidence value as a match.

31. The computer system of claim 25 wherein enabling presentation comprises enabling presentation of information including both sponsored-content with content other than sponsored content.