Image-based search engine for mobile phones with camera

• US 7,565,139 B2
• Filed: 05/13/2005
• Issued: 07/21/2009
• Est. Priority Date: 02/20/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Neven, Hartmut Sr., Neven, Hartmut
• Primary Examiner(s): Tran; Tuan A
• Application Number:
  US11/129,034
• Publication Number:
  US 20060012677A1
• Time in Patent Office: 1,530 Days
• Field of Search: 4554141- 4, 455418-420, 455/458, 455/550.1, 4555561- 2, 455 301- 306, 382/118, 382/182, 382/229, 382/181, 707/104.1, 707 3- 10, 707200-204, 348 1401- 1402, 345418-419
• US Class Current: 455/414.3
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/242 Division of the character sequences into groups prior to recognition; Selection of dictionaries

Abstract

An image-based information retrieval system is disclosed that includes a mobile telephone and a remote server. The mobile telephone has a built-in camera and a communication link for transmitting an image from the built-in camera to the remote server. The remote server has an optical character recognition engine for generating a first confidence value based on an image from the mobile telephone, an object recognition engine for generating a second confidence value based on an image from the mobile telephone, a face recognition engine for generating a third confidence value based on an image from the mobile telephone, and an integrator module for receiving the first, second, and third confidence values and generating a recognition output.

Claims

1. A method for updating object representations in an image database, the method comprising:

receiving an image of an object from a client device associated with a user, the receiving of the image constituting a first user query for information related to the object;

recognizing the object using an image recognition engine and an image database, the image database storing object identifiers in association with objects;

associating, from the image database, an object identifier with the recognized object;

transmitting to the client device associated with the user, information related to the recognized object;

receiving an indication of whether the user accepted the transmitted information as being relevant to the object; and

responsive to a computed confidence measure of the recognition of the object exceeding a given threshold, storing the image of the object in the image database in association with the object identifier;

wherein the confidence measure is computed at least in part based on the indication that the user accepted the transmitted information as being relevant to the object.

2. The method of claim 1, wherein the confidence measure is computed at least based in part on an amount of time that the client device displays the transmitted information.

3. The method of claim 1, wherein the confidence measure is computed at least based in part on whether a second user query for information related to the object is received within a predetermined amount of time of the first user query.

4. The method of claim 1, further comprising identifying a web page address related to the recognized object and storing the web page address in association with the object identifier.

5. The method of claim 1, further comprising

identifying an audio stream related to the recognized object and storing the audio stream in association with the object identifier.

6. The method of claim 1, wherein the recognized object comprises text, the method further comprising translating the text and storing the translated text in association with the object identifier.

7. The method of claim 1, wherein recognizing the object comprises at least one of:

performing optical character recognition, performing object recognition, and performing facial recognition.

8. The method of claim 7, wherein performing object recognition comprises performing rigid texture object recognition.

9. The method of claim 7, wherein performing object recognition comprises performing articulate object recognition.

10. The method of claim 1, wherein the image of the object is obtained using a built-in camera of the client device.

11. A recognition computer server system for updating object representations in an image database, the recognition computer server system adapted to perform actions comprising:

receiving an image of an object from a client device associated with a user, the receiving of the image constituting a first user query for information related to the object;

recognizing the object using an image recognition engine and an image database, the image database storing object identifiers in association with objects;

associating, from the image database, an object identifier with the recognized object;

transmitting to the client device associated with the user, information related to the recognized object;

receiving an indication of whether the user accepted the transmitted information as being relevant to the object; and

responsive to a computed confidence measure of the recognition of the object exceeding a given threshold, storing the image of the object in the image database in association with the object identifier;

wherein the confidence measure is computed at least in part based on the indication that the user accepted the transmitted information as being relevant to the object.

12. The recognition computer server system of claim 11, wherein the confidence measure is computed at least based in part on an amount of time that the client device displays the transmitted information.

13. The recognition computer server system of claim 11, wherein the confidence measure is computed at least based in part on whether a second user query for information related to the object is received within a predetermined amount of time of the first user query.

14. The recognition computer server system of claim 11, the actions of the recognition computer server system further comprising identifying a web page address related to the recognized object and storing the web page address in association with the object identifier.

15. The recognition computer server system of claim 11, the actions of the recognition computer server system further comprising identifying an audio stream related to the recognized object and storing the audio stream in association with the object identifier.

16. The recognition computer server system of claim 11, wherein the recognized object comprises text, the method further comprising translating the text and storing the translated text in association with the object identifier.

17. The recognition computer server system of claim 11, wherein recognizing the object comprises at least one of:

performing optical character recognition, performing object recognition, and performing facial recognition.

18. The recognition computer server system of claim 17, wherein performing object recognition comprises performing rigid texture object recognition.

19. The recognition computer server system of claim 17, wherein performing object recognition comprises performing articulate object recognition.

20. The recognition computer server system of claim 11, wherein the image of the object is obtained using a built-in camera of the client device.