Query-by-image search and retrieval system

• US 7,702,681 B2
• Filed: 06/29/2005
• Issued: 04/20/2010
• Est. Priority Date: 06/29/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Macbeth, Randall J., Watson, Eric B., Brewer, Brett D., Whyte, Nicholas A.
• Primary Examiner(s): Alam; Hosain T
• Assistant Examiner(s): Tang; Jieying
• Application Number:
  US11/170,267
• Publication Number:
  US 20070005571A1
• Time in Patent Office: 1,756 Days
• Field of Search: 707/3, 707/5, 707/6, 707/104.1, 715/230, 382/181
• US Class Current: 707/748
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/944 Business related

Abstract

An image query system, and associated method and computer program product are disclosed that enhance and accelerate image-based querying. To this end, the present invention presents a digital image query with implict and explicit associated metadata and retrieval of visually similar images or other pertinent information from a large collection of images stored on local, networked, or web-linked repositories.

Claims

1. A method for searching a database through an image query, comprising the steps of:

inputting a digital image stored in a file to a first workstation having a data entry device, wherein implicit data associated with the image is stored in the same file as the image, and wherein the implicit data is permanent, non-modifiable information that is attached into the image as electronic pixels selected in a pattern that displays the implicit data in a manner that contrasts with the image;

annotating explicitly said image with explicit data using the data entry device to thereby produce an annotated image wherein explicit data is at least a factual description of the image;

searching the database of candidate images with the annotated image, wherein searching the database comprises comparing the image to candidate images in the database using photomatching technology, comparing implicit data associated with the image to implicit data associated with the candidate images in the database, and comparing explicit data associated with the image to explicit data associated with the candidate images in the database;

searching one or more online encyclopedias and one or more mapping services with the implicit and explicit data stored in the file with the annotated image; and

generating a result set including one or more records wherein the one or more records are ranked according to a ranking criteria.

2. The method of claim 1, further including the step of capturing the image with a camera.

3. The method in accordance with claim 2, wherein said implicit data includes a geocode location for the image.

4. The method of claim 1, wherein the step of annotating explicitly said image with explicit data comprises annotating the image with audio data.

10. The method of claim 1, further comprising after searching the database of candidate images with the annotated images, adding the annotated image to the database of candidate images.

5. An image retrieval system comprising:

a workstation comprising a digital image stored in the workstation, wherein implicit data associated with the image is stored in the same file as the image, wherein the implicit data is permanent, non-modifiable information that is attached into the image as electronic pixels selected in a pattern that displays the implicit data in a manner that contrasts with the image, and wherein the implicit data includes a geocode location for the image;

entry means for annotating the image with explicit data to create a query image, wherein explicit data is at least a factual description of the image, and wherein the implicit data associated with the image is also associated with the query image;

a database of candidate images with explicit data;

image comparison means for comparing the query image with the candidate images using photomatching technology establish respective image similarities between the query image and the respective candidate images;

metadata comparison means for comparing the implicit data of the query image to the implicit data of the candidate images;

metadata comparison means for comparing the explicit data of the query image to the explicit data of the candidate images;

subquery means for parsing the implicit and explicit data of the query image to create subqueries that identify sponsored advertisements directed to a location or item associated with the query image; and

presentation means for presenting at least a candidate image with a largest image and data similarity based on a ranking criteria and one or more identified sponsored advertisements.

6. The system of claim 5, wherein the entry means is a data tag application.

7. The system of claim 5, wherein annotating the image with explicit data comprises annotating the image with audio data.

8. One or more computer-readable storage media having computer useable instructions stored thereon for implementing a content and metadata based image query, comprising:

a first set of instruction codes for annotating a digital image with explicit metadata, wherein explicit metadata is at least a factual description of the image, wherein implicit data associated with the image is stored in the same file as the image, and wherein the implicit data is permanent, non-modifiable information that is attached into the image as electronic pixels selected in a pattern that displays the implicit data in a manner that contrasts with the image;

a second set of instruction codes for comparing the annotated image with a candidate image from a database of candidate images by using photomatching technology, comparing the implicit data associated with the candidate image, and comparing the explicit metadata of the annotated image with explicit data associated with the candidate image,

wherein after comparing the annotated image with a candidate image, the annotated image is added to the database of candidate images;

a third set of instruction codes for parsing the implicit and explicit data of the annotated image to create subqueries, using the subqueries to search one or more online encyclopedias and one or more mapping services, and identifying sponsored advertisements directed to a location or item associated with the annotated image; and

a fourth set of instruction codes for implementing a ranking criteria to display one or more candidate images with relevant results.

9. The media of claim 8, wherein annotating the image with explicit metadata comprises annotating the image with audio data.