Photo-based mobile deixis system and related techniques

• US 20050162523A1
• Filed: 01/22/2004
• Published: 07/28/2005
• Est. Priority Date: 01/22/2004
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Darrell, Trevor J., Tollmar, Konrad, Yeh, Tom
• Granted Patent: US 7,872,669 B2
• Time in Patent Office: Days
• US Class Current: 348/211.200
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A mobile deixis device includes a camera to capture an image and a wireless handheld device, coupled to the camera and to a wireless network, to communicate the image with existing databases to find similar images. The mobile deixis device further includes a processor, coupled to the device, to process found database records related to similar images and a display to view found database records that include web pages including images. With such an arrangement, users can specify a location of interest by simply pointing a camera-equipped cellular phone at the object of interest and by searching an image database or relevant web resources, users can quickly identify good matches from several close ones to find an object of interest.

Claims

1. A mobile deixis device comprising:

(a) a camera to capture an image;

(b) a wireless communication device, coupled to the camera and to a wireless network, to communicate the image with existing databases to find similar images; and

(c) a processor, coupled to the wireless communication device, to process found database records related to similar images.

2. The mobile deixis device as recited in claim 1 comprising a display to view found database records and the found database records comprising web pages including images.

3. The mobile deixis device as recited in claim 1 comprising:

(a) a storage medium coupled to the processor; and

(b) a plurality of programs stored in the storage medium operative to interact with the processor and the wireless communication device to control the operation of the mobile deixis device, comprising;

(i) a first program of the plurality of programs stored on the storage medium being operative to interact with the processor to capture the image from the camera;

(ii) a second program of the plurality of programs stored on the storage medium being operative to interact with the processor to communicate with at least one database to find a similar image similar to the captured image; and

(iii) a third program of the plurality of programs stored on the storage medium being operative to interact with the processor to provide to a display a plurality of similar images and maintaining an associated hyperlink for each similar image.

4. The mobile deixis device as recited in claim 3 wherein the second program of the plurality of programs stored on the storage medium further comprises a subprogram stored on the storage medium being operative to interact with the processor to communicate with at least one server database to cause said server database to search further databases for a similar image similar to the captured image.

5. The mobile deixis device as recited in claim 1 comprising:

(a) a storage medium coupled to the processor; and

(b) a plurality of programs stored in the storage medium operative to interact with the processor and the wireless communication device to control the operation of the mobile deixis device, comprising;

(i) a first program of the plurality of programs stored on the storage medium being operative to interact with the processor to capture the image from the camera;

(ii) a second program of the plurality of programs stored on the storage medium being operative to interact with the processor to communicate with at least one computer having a database of images to cause the at least one computer to search the database of images to find a similar image similar to the captured image; and

(iii) a third program of the plurality of programs stored on the storage medium being operative to interact with the processor to provide to a display of a plurality of similar images and maintaining an associated hyperlink for each similar image.

6. The mobile deixis device as recited in claim 1 comprising:

(a) a storage medium coupled to the processor; and

(b) a plurality of programs stored in the storage medium operative to interact with the processor and the wireless communication device to control the operation of the mobile deixis device, comprising;

(i) a first program of the plurality of programs stored on the storage medium being operative to interact with the processor to capture the image from the camera;

(ii) a second program of the plurality of programs stored on the storage medium being operative to interact with the processor to communicate with at least one computer with web pages having images to search the images to find a similar image similar to the captured image; and

(iii) a third program of the plurality of programs stored on the storage medium being operative to interact with the processor to provide to a display of a plurality of similar images and maintaining an associated hyperlink for each similar image.

7. The mobile deixis device as recited in claim 3 wherein the at least one database includes a web site dispersed within the Internet and the plurality of programs stored in the storage medium further comprises:

(i) a fourth program of the plurality of programs stored on the storage medium being operative to interact with the processor to identify any keywords linked to each similar image; and

(ii) a fifth program of the plurality of programs stored on the storage medium being operative to interact with the processor to initiate a further search using the keywords to find additional similar images.

8. The mobile deixis device as recited in claim 3 further comprising:

a global positioning system (GPS) receiver to identify the geographical location of the mobile deixis device; and

wherein the plurality of programs stored in the storage medium operative to interact with the processor and the mobile communication device comprises;

an additional program of the plurality of programs stored on the storage medium being operative to interact with the processor to eliminate any similar images that are known not to be located in the geographical location of the mobile deixis device.

9. A system to provide location awareness services comprising:

(a) a handheld device comprising;

(i) a camera to capture an image of an location;

(ii) a mobile communication device, coupled to the camera and to a wireless network, to communicate the image with existing computer files to find similar images; and

(iii) a user interface, coupled to the mobile communication device, to communicate to an user any results of found similar images; and

(b) a computer network including the wireless network and a wired network;

(c) a plurality of computers, each computer have a plurality of computer files and connected to the computer network, at least one of the computer files having an image similar to the captured image and when viewed includes associated text describing an object in the image.

10. The system to provide location awareness services as recited in claim 9 wherein the user interface comprises a display.

11. The system to provide location awareness services as recited in claim 9 wherein the user interface comprises a computer connection.

12. The system to provide location awareness services as recited in claim 9 wherein the associated text describes features of the object in the image including geographical location of the object.

13. The system to provide location awareness services as recited in claim 9 wherein the associated text describes features of the object in the image including a description and historical facts regarding the object.

14. The system to provide location awareness services as recited in claim 9 wherein the associated text includes a uniform resource locator (URL).

15. The system to provide location awareness services as recited in claim 9 wherein at least one of the computers includes a plurality of computer files having images of locations of interest located within a predetermined radius about a geographical location.

16. The system to provide location awareness services as recited in claim 15 wherein the computer having a plurality of computer files having images of objects of interest located within a predetermined radius about a geographical location was previously trained to find common objects known to be of interest.

17. The system to provide location awareness services as recited in claim 9 wherein at least one of the computers includes at least one computer file having an image of an object of known interest and an associated image of an object of less recognized interest within a predetermined radius about a geographical location of the known interest object to aid a user in finding the object of less recognized interest.

18. The system to provide location awareness services as recited in claim 9 wherein at least one of the computers includes at least one computer file having an image of an object of known interest and an associated image of an object of less recognized interest within the field of view of the known interest object to aid a user in finding the object of less recognized interest.

19. The system to provide location awareness services as recited in claim 9 wherein at least one of the computers includes a capability to search other computers having at least one computer file having text matching the associated text describing the object in the an image.

20. The system to provide location awareness services as recited in claim 19 wherein at least one computer file having text matching the associated test describing the object in the image is communicated to the handheld device.

21. The system to provide location awareness services as recited in claim 19 wherein at least one computer file having text matching the associated test describing the object in the image is communicated to the computer that initiated the search.

22. The system to provide location awareness services as recited in claim 21 wherein the computer that initiated the search is capable of comparing the original image with images returned in the computer file having text matching the associated text describing the object in the image.

23. The system to provide location awareness services as recited in claim 15 further comprising:

a global positioning system (GPS) receiver to identify the geographical location of the mobile communication device to help eliminate non-useful images.

24. The system to provide location awareness services as recited in claim 19 wherein at least one of the computers includes at least one computer file having an image of an object of known interest and an associated image of an object of less recognized interest within the field of view of the known interest object to aid a user in finding the object of less recognized interest all located within a predetermined radius about a geographical location.

25. A storage medium comprising:

a first computer readable program code stored on the storage medium being operative to interact with a processor in a handheld device to capture an image from a camera;

a second computer readable program code stored on the storage medium being operative to interact with the processor to communicate with multiple data files and to search said data files for a similar image similar to the captured image; and

a third computer readable program code stored on the storage medium being operative to interact with the processor to provide to an user interface a plurality of similar images and maintaining an associated hyperlink for each similar image.

26. A method for identifying a location comprising the steps of:

(i) providing a database of images, each image having an associated URL that includes said image and a description of the image;

(ii) comparing an image of an unknown location with images from the database of images and providing a list of possible matching images; and

(iii) reviewing the images in the list of possible matching images until the correct location is identified.

27. The method for identifying a location as recited in claim 26 wherein the comparing step includes comparing at least one of energy spectrum data, color histogram data, primitive filter data, and local invariant data.

28. The method for identifying a location as recited in claim 26 wherein the comparing step comprises at least one of the techniques including a least square matching technique, a normalizing the image technique, an eigen value technique, a matching histogram of image feature technique and an image matching engine with transformation technique.