Image base inquiry system for search engines for mobile telephones with integrated camera

• US 8,421,872 B2
• Filed: 02/20/2004
• Issued: 04/16/2013
• Est. Priority Date: 02/20/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Neven, Hartmut Sr.
• Primary Examiner(s): GILES, NICHOLAS G
• Application Number:
  US10/783,378
• Publication Number:
  US 20050185060A1
• Time in Patent Office: 3,343 Days
• Field of Search: 348/161, 348/231.2, 348/222.1, 382/165, 382/162, 382/170, 382/181, 382/182, 382/118, 382/305, 382/306, 382/209, 382/218, 382/224
• US Class Current: 348/222.1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: H04N 1/00204 with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32112 in a separate computer file, document page or paper sheet, e.g. a fax cover sheet H04N1/32122 takes precedence
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices

Abstract

An increasing number of mobile telephones and computers are being equipped with a camera. Thus, instead of simple text strings, it is also possible to send images as queries to search engines or databases. Moreover, advances in image recognition allow a greater degree of automated recognition of objects, strings of letters, or symbols in digital images. This makes it possible to convert the graphical information into a symbolic format, for example, plain text, in order to then access information about the object shown.

Claims

1. A system for image-based information retrieval from search engines, characterized by a) a terminal with a built-in camera that is connected to a remote data transmission network;

b) a server computer on which an object recognition program is running, which analyses images sent to it and provides them with a symbolic indexing;

c) a search engine that uses the indexed image to find information about the image and sends it back to the terminal.

2. The system as described under 1) that is designed for mobile telephones or portable computers that have a built-in camera.

3. A city or museum guide that uses the system described under 2) to provide a user with information about objects of which he or she has previously taken a picture.

4. The system as described under 3) in which positioning information is also used to appropriately limit the image recognition system.

5. The system as described under 2) that provides product information about products that have been previously photographed with the mobile camera.

6. The system as described under 2) in which the image recognition system is also able to recognize text characters or symbols.

7. The system as described under 2) in which the system is, in particular, able to recognize faces.

8. The system as described under 2) that is used to provide the user with additional information about advertising billboards.

9. An electronic user handbook that uses a system as described under 2) to quickly navigate access to corresponding sections of the handbook.

10. The system as described under 2) that allows providers of information to independently make new entries in the image processing system for the purpose of allowing their data to be retrieved by means of image entry.

11. A computer implemented system for image-based searching, comprising:

a computer server, communicatively coupled with a network, that receives an input image from a user device communicatively coupled with the network;

an image recognition system executed by the computer server and adapted to;

determine a plurality of graphical attributes of the input image;

match the input image to a reference image from a plurality of reference images stored in a storage medium, based on the plurality of graphical attributes of the input image and the reference images, each of the reference images having an associated symbolic identifier; and

associate a symbolic identifier to the input image based on the symbolic identifier associated with the matching reference image;

a search engine executed by the computer server and adapted to receive a query and to retrieve a set of search results associated with the query; and

a query processing system executed by the computer server and adapted to;

receive the symbolic identifier of the input image from the image recognition system;

provide the symbolic identifier to the search engine as a query, and to receive a set of search results associated with the symbolic identifier; and

transmit, via the network, a plurality of the search results to the user device.

12. The system of claim 11, wherein the user device comprises a mobile telephone having an integrated camera.

13. The system of claim 11, wherein:

the server receives a geographic location of the user device in association with the input image; and

the image recognition system is further adapted to match the input image to a reference image from the plurality of reference images based on the geographic location of the user device.

14. The system of claim 11, wherein the image recognition system further includes a character recognition system.

15. The system of claim 11, wherein the image recognition system further includes a facial recognition system.

16. The system of claim 11, wherein the image recognition system is further adapted to:

receive a plurality of images from the user device;

store the received images as reference images; and

match an input image subsequently received from the user device to at least one of the reference images received from the user device.

17. A computer implemented method for image-based searching, comprising:

receiving at a computer server, an input image from a user device remotely located from the server;

providing from the computer server the input image to an image recognition system;

receiving at the computer server from the image recognition system a symbolic identifier associated with the input image;

providing from the computer server the symbolic identifier to a search engine as a query;

receiving at the computer server from the search engine a set of search results associated with the symbolic identifier; and

transmitting from the computer server a plurality of the search results to the user device.

18. A computer implemented method for image-based searching at a computer server, the method comprising:

receiving an input image from a user device remotely located from the server;

determining a plurality of graphical attributes represented in the input image;

matching the input image to a reference image from a plurality of reference images stored in a storage medium, based on the plurality of graphical attributes of the input image and the reference images, each of the reference images having an associated symbolic identifier; and

associating a symbolic identifier to the input image based on the symbolic identifier associated with the matching reference image;

processing the symbolic identifier as search query to retrieve, from a search engine, a set of search results associated with the symbolic identifier; and

transmitting a plurality of the search results to the user device.

19. The method of claim 18, wherein the user device comprises a mobile telephone having an integrated camera.

20. The method of claim 18, wherein the image recognition system is further adapted to:

receive a plurality of reference images from the user device;

store the received images as reference images; and

match an input image subsequently received from the user device to at least one of the reference images received from the user device.

21. The method of claim 18, further comprising receiving a geographic location of the user device through the network.

22. The method of claim 21, wherein the image recognition system is further adapted to match the input image to a reference image from the plurality of reference images based on the geographic location of the user device.

23. The method of claim 18, wherein the image recognition system further includes a character recognition system.

24. The method of claim 18, wherein the image recognition system further includes a facial recognition system.

25. The method of claim 18, wherein the image recognition system is further adapted to enable transmission of reference images, for use by the image recognition system, to the storage medium.

26. The method of claim 18, wherein selecting a matching reference image from a plurality of reference images stored in a storage medium comprises:

determining the graphical attributes in the input image represented by a plurality of trained attribute detectors;

aggregating a plurality of confidence values received from the plurality of trained attribute detectors; and

determining the matching reference image where the aggregated plurality of confidence values exceed a predetermined threshold value.

27. The method of claim 18, wherein the search results comprise links to websites, contact information, product information, translations of recognized characters, and other information related to the input image.

28. A computer implemented method for image-based searching of product information, comprising:

receiving an input image from a user device remotely located from the computer server;

processing the input image of a manufactured product on an image recognition system to obtain a symbolic identifier identifying the manufactured product in the input image, the symbolic identifier comprising at least one of a product name or a product identification number, or a product code;

providing the symbolic identifier associated with the input image to the search engine as a query;

receiving a set of search results associated with the symbolic identifier, the search results including at least one document descriptive of the manufactured product in the input image; and

transmitting via the network, a plurality of the search results to the user device.

29. A computer implemented method for image-based identification of buildings, comprising:

receiving an input image of a building from a user device remotely located from the computer server;

processing the input image of the building on an image recognition system executed by the computer server to obtain a symbolic identifier identifying the building in the input image, the symbolic identifier comprising at least one of a building name or a building location;

providing the symbolic identifier associated with the input image to a search engine as a query;

receiving a set of search results associated with the symbolic identifier, the search results including at least one document descriptive of the building in the input image; and

transmitting via the network, a plurality of the search results to the user device.

30. A computer implemented method for image-based language translation, comprising:

receiving an input image from a user device remotely located from the computer server;

processing the input image on a character recognition system, executed by the computer server to obtain text data indicative of the text in the input image, the text data in a first language; and

providing the text data and an indication of a user-specified second language to a translation system;

receiving from the translation system a translation of the text data into a second language; and

transmitting, via the network, the translation to the user device.

31. A computer implemented method for image-based searching of human faces, comprising:

receiving an input image of a human face on a computer server communicatively coupled with a network from a user device communicatively coupled with the network; and

processing the input image of the human face on a facial recognition system to obtain a symbolic identifier identifying the human face in the input image, the symbolic identifier comprising a name of the person having the human face in the input image;

providing the symbolic identifier associated with the input image to a search engine as a query;

receiving a set of search results associated with the symbolic identifier, the search results including at least one document descriptive of the person in the input image; and

transmitting via the network, a plurality of the search results to the user device.